[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:25 dist-test-slave-zt01 krb5kdc[15181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:25 dist-test-slave-zt01 krb5kdc[15181](info): set up 2 sockets
Jun 20 06:11:25 dist-test-slave-zt01 krb5kdc[15181](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 06:11:26.792716 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:26 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:26.910697 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34617
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:34617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:27.016288 15194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:27.016558 15194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:27.016630 15194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:27.020500 15194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:27.020581 15194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:27.020619 15194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:27.020677 15194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:27.020728 15194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:27.025686 15194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:34617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:27.026628 15194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:27.027541 15194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:27.036582 15202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.036712 15200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.036661 15199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:27.036794 15194 server_base.cc:1034] running on GCE node
I20260620 06:11:27.037281 15194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:27.038304 15194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:27.039453 15194 hybrid_clock.cc:648] HybridClock initialized: now 1781935887039427 us; error 38 us; skew 500 ppm
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:27.042130 15194 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:27.042979 15194 webserver.cc:460] Webserver started at http://127.14.209.190:35283/ using document root <none> and password file <none>
I20260620 06:11:27.043545 15194 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:27.043632 15194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:27.043851 15194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:27.045740 15194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "2ca2319f687148b0a843a5295a0f560e"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "f455a91ec93115b840bf10f7623d25d2"
server_key_iv: "ef5692d54b23b84ac8b7903e3094c226"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.046289 15194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "2ca2319f687148b0a843a5295a0f560e"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "f455a91ec93115b840bf10f7623d25d2"
server_key_iv: "ef5692d54b23b84ac8b7903e3094c226"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.049822 15194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:11:27.052192 15209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.053396 15194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:27.053539 15194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "2ca2319f687148b0a843a5295a0f560e"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "f455a91ec93115b840bf10f7623d25d2"
server_key_iv: "ef5692d54b23b84ac8b7903e3094c226"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.053644 15194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:27.071604 15194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:27.074350 15194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:27.074549 15194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:27.081295 15194 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:34617
I20260620 06:11:27.081310 15261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:34617 every 8 connection(s)
I20260620 06:11:27.082157 15194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:27.084832 15262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:27.086544 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15194
I20260620 06:11:27.086694 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:27.088258 15174 external_mini_cluster.cc:1428] Setting key de7f8334e31b3f926a953add48170ff8
I20260620 06:11:27.091370 15262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e: Bootstrap starting.
I20260620 06:11:27.093559 15262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:27.094234 15262 log.cc:824] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:27.096159 15262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e: No bootstrap required, opened a new log
I20260620 06:11:27.100059 15262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2319f687148b0a843a5295a0f560e" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34617 } }
I20260620 06:11:27.100293 15262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:27.100343 15262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca2319f687148b0a843a5295a0f560e, State: Initialized, Role: FOLLOWER
I20260620 06:11:27.100759 15262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2319f687148b0a843a5295a0f560e" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34617 } }
I20260620 06:11:27.100894 15262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:27.100955 15262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:27.101074 15262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:27.102088 15262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2319f687148b0a843a5295a0f560e" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34617 } }
I20260620 06:11:27.102416 15262 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca2319f687148b0a843a5295a0f560e; no voters:
I20260620 06:11:27.102752 15262 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:27.102854 15267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:27.103104 15267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [term 1 LEADER]: Becoming Leader. State: Replica: 2ca2319f687148b0a843a5295a0f560e, State: Running, Role: LEADER
I20260620 06:11:27.103412 15267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2319f687148b0a843a5295a0f560e" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34617 } }
I20260620 06:11:27.103904 15262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:27.105000 15268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca2319f687148b0a843a5295a0f560e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2319f687148b0a843a5295a0f560e" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34617 } } }
I20260620 06:11:27.105142 15268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:27.105405 15269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca2319f687148b0a843a5295a0f560e. Latest consensus state: current_term: 1 leader_uuid: "2ca2319f687148b0a843a5295a0f560e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2319f687148b0a843a5295a0f560e" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34617 } } }
I20260620 06:11:27.105531 15269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:27.105496 15276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:27.109170 15276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:27.114415 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.091885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.14.209.190:34617)
0620 06:11:27.092354 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:27.092377 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:27.093348 (+ 971us) server_negotiation.cc:407] Connection header received
0620 06:11:27.096753 (+ 3405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:27.096773 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:27.097118 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:27.097443 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:27.099438 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.100890 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.102603 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.103324 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.104815 (+ 1491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.108306 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:27.108312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:27.108328 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:27.108357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:27.110663 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.111267 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.111274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.111280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.111417 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.111820 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.111823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.111825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.112331 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:27.112530 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:27.113143 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 06:11:27.113405 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260620 06:11:27.116791 15276 catalog_manager.cc:1347] Generated new cluster ID: 0ba8b7677f97450d86d1da39bad1ef16
I20260620 06:11:27.116883 15276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:27.127750 15276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:27.128803 15276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:27.136442 15276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca2319f687148b0a843a5295a0f560e: Generated new TSK 0
I20260620 06:11:27.137087 15276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:27.172117 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34617
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:27.280382 15288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:27.280647 15288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:27.280723 15288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:27.284614 15288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:27.284690 15288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:27.284783 15288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:27.290164 15288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34617
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:27.291172 15288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:27.292140 15288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:27.301687 15296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.301714 15293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.301709 15294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:27.301752 15288 server_base.cc:1034] running on GCE node
I20260620 06:11:27.302255 15288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:27.302842 15288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:27.304052 15288 hybrid_clock.cc:648] HybridClock initialized: now 1781935887304028 us; error 39 us; skew 500 ppm
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:27.307258 15288 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:27.308037 15288 webserver.cc:460] Webserver started at http://127.14.209.129:45361/ using document root <none> and password file <none>
I20260620 06:11:27.308519 15288 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:27.308574 15288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:27.308722 15288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:27.310611 15288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "87b95dfa338149ab86be14df52740a51"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "39dc9dbc4f830d0ed3c772ca841e434f"
server_key_iv: "dc559c33a3989c75386c571e13109628"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.311179 15288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "87b95dfa338149ab86be14df52740a51"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "39dc9dbc4f830d0ed3c772ca841e434f"
server_key_iv: "dc559c33a3989c75386c571e13109628"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.314574 15288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:27.316807 15303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.317938 15288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:27.318074 15288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "87b95dfa338149ab86be14df52740a51"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "39dc9dbc4f830d0ed3c772ca841e434f"
server_key_iv: "dc559c33a3989c75386c571e13109628"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.318177 15288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:27.334213 15288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:27.336944 15288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:27.337141 15288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:27.337663 15288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:27.338426 15288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:27.338495 15288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.338558 15288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:27.338606 15288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.347024 15288 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:40657
I20260620 06:11:27.347132 15416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:40657 every 8 connection(s)
I20260620 06:11:27.347932 15288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:27.358033 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15288
I20260620 06:11:27.358144 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:27.358457 15174 external_mini_cluster.cc:1428] Setting key 13f6b79665a92724f9ed58e0ae346965
I20260620 06:11:27.361912 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.349602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:55697 (local address 127.14.209.190:34617)
0620 06:11:27.349732 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:27.349736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:27.350671 (+ 935us) server_negotiation.cc:407] Connection header received
0620 06:11:27.351598 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:27.351602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:27.351660 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:27.351772 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:27.352822 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.353379 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.354472 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.354705 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.354831 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.357351 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:27.357356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:27.357362 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:27.357407 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:27.359021 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.360116 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.360118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.360120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.360325 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.360674 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.360677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.360680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.360921 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:27.361026 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:27.361513 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 06:11:27.361654 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:11:27.362942 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.349915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:34617 (local address 127.14.209.129:55697)
0620 06:11:27.350533 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:27.350568 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:27.351405 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:27.351927 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:27.351934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:27.352252 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:27.352679 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:27.352691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.353512 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.353520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:27.354350 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:27.354358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.354864 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.354869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:27.355033 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.355666 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:27.355682 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:27.357168 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:27.359543 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:27.359549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:27.359561 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:27.359890 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:27.360435 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:27.360438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:27.360439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:27.360565 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:27.361046 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:27.361051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:27.361259 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 06:11:27.361418 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 06:11:27.361648 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":117,"threads_started":1}
I20260620 06:11:27.364229 15417 heartbeater.cc:347] Connected to a master server at 127.14.209.190:34617
I20260620 06:11:27.364457 15417 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:27.364897 15417 heartbeater.cc:511] Master 127.14.209.190:34617 requested a full tablet report, sending...
I20260620 06:11:27.366503 15226 ts_manager.cc:194] Registered new tserver with Master: 87b95dfa338149ab86be14df52740a51 (127.14.209.129:40657)
I20260620 06:11:27.368098 15226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:55697
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:27.386052 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34617
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:27.493240 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:27.493469 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:27.493541 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:27.498147 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:27.498202 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:27.498325 15422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:27.503844 15422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34617
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:27.504853 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:27.505824 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:27.514823 15430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.514848 15428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.514827 15427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:27.515234 15422 server_base.cc:1034] running on GCE node
I20260620 06:11:27.515601 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:27.516219 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:27.517385 15422 hybrid_clock.cc:648] HybridClock initialized: now 1781935887517359 us; error 40 us; skew 500 ppm
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:27.520088 15422 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:27.520941 15422 webserver.cc:460] Webserver started at http://127.14.209.130:39687/ using document root <none> and password file <none>
I20260620 06:11:27.521447 15422 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:27.521548 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:27.521754 15422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:27.523717 15422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "fafdc565c3124829bca90965ba0db4b7"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "82413a02e437472b2a255ba93bb7e7c5"
server_key_iv: "d3c981f9d1629418f32a35e86f1194c7"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.524255 15422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "fafdc565c3124829bca90965ba0db4b7"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "82413a02e437472b2a255ba93bb7e7c5"
server_key_iv: "d3c981f9d1629418f32a35e86f1194c7"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.527704 15422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:27.529938 15437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.531032 15422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:27.531222 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "fafdc565c3124829bca90965ba0db4b7"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "82413a02e437472b2a255ba93bb7e7c5"
server_key_iv: "d3c981f9d1629418f32a35e86f1194c7"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.531332 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:27.546983 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:27.550029 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:27.550215 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:27.550738 15422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:27.551543 15422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:27.551620 15422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.551688 15422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:27.551723 15422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.560565 15422 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:42605
I20260620 06:11:27.560592 15550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:42605 every 8 connection(s)
I20260620 06:11:27.561385 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:27.571595 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15422
I20260620 06:11:27.571715 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:27.571972 15174 external_mini_cluster.cc:1428] Setting key a86b1028ce1d6d01000f7183119dcdef
I20260620 06:11:27.573827 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.563088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:59731 (local address 127.14.209.190:34617)
0620 06:11:27.563207 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:27.563211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:27.564152 (+ 941us) server_negotiation.cc:407] Connection header received
0620 06:11:27.564926 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:27.564929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:27.564973 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:27.565050 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:27.566090 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.566599 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.567716 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.567950 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.568030 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.570253 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:27.570256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:27.570258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:27.570281 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:27.571657 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.572254 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.572258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.572260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.572346 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.572718 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.572722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.572724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.572905 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:27.572983 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:27.573560 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 06:11:27.573671 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:11:27.574527 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.563407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:34617 (local address 127.14.209.130:59731)
0620 06:11:27.564015 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:27.564051 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:27.564737 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:27.565189 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:27.565195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:27.565501 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:27.565948 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:27.565961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.566699 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.566701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:27.567610 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:27.567619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.568070 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.568073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:27.568201 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.568747 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:27.568763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:27.570106 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:27.571800 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:27.571806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:27.571820 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:27.572157 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:27.572451 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:27.572454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:27.572456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:27.572627 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:27.573009 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:27.573014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:27.573301 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 06:11:27.573504 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 06:11:27.573788 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":105,"threads_started":1}
I20260620 06:11:27.575625 15551 heartbeater.cc:347] Connected to a master server at 127.14.209.190:34617
I20260620 06:11:27.575862 15551 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:27.576323 15551 heartbeater.cc:511] Master 127.14.209.190:34617 requested a full tablet report, sending...
I20260620 06:11:27.577385 15226 ts_manager.cc:194] Registered new tserver with Master: fafdc565c3124829bca90965ba0db4b7 (127.14.209.130:42605)
I20260620 06:11:27.577962 15226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:59731
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:27.598865 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34617
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:27.707428 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:27.707645 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:27.707685 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:27.711526 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:27.711572 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:27.711669 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:27.716691 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34617
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:27.717630 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:27.718554 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:27.727725 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.727734 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:27.727728 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:27.728108 15556 server_base.cc:1034] running on GCE node
I20260620 06:11:27.728529 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:27.729086 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:27.730273 15556 hybrid_clock.cc:648] HybridClock initialized: now 1781935887730266 us; error 29 us; skew 500 ppm
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:27.732949 15556 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:27.733809 15556 webserver.cc:460] Webserver started at http://127.14.209.131:40989/ using document root <none> and password file <none>
I20260620 06:11:27.734364 15556 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:27.734457 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:27.734680 15556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:27.736737 15556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "2c29575c0126446da5afdd0bb72638e6"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "2a06e80d5c4a9781662e7e06cf348413"
server_key_iv: "45339402a08cf939a893b947b1b01867"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.737293 15556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "2c29575c0126446da5afdd0bb72638e6"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "2a06e80d5c4a9781662e7e06cf348413"
server_key_iv: "45339402a08cf939a893b947b1b01867"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.740980 15556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:27.743219 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.744335 15556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:27.744470 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "2c29575c0126446da5afdd0bb72638e6"
format_stamp: "Formatted at 2026-06-20 06:11:27 on dist-test-slave-zt01"
server_key: "2a06e80d5c4a9781662e7e06cf348413"
server_key_iv: "45339402a08cf939a893b947b1b01867"
server_key_version: "encryptionkey@0"
I20260620 06:11:27.744575 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:27.763628 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:27.766623 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:27.766805 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:27.767387 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:27.768119 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:27.768187 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.768251 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:27.768301 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:27.776859 15556 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:41493
I20260620 06:11:27.776937 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:41493 every 8 connection(s)
I20260620 06:11:27.777662 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:27.784384 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15556
I20260620 06:11:27.784494 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:27.784745 15174 external_mini_cluster.cc:1428] Setting key 002cc2277660bdab4c04542ce51eae39
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:27.789855 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.779368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:37691 (local address 127.14.209.190:34617)
0620 06:11:27.779509 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:27.779513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:27.780504 (+ 991us) server_negotiation.cc:407] Connection header received
0620 06:11:27.781245 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:27.781247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:27.781292 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:27.781362 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:27.782374 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.782883 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.784001 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.784220 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.784294 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.787045 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:27.787048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:27.787086 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:27.787116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:27.788290 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.788715 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.788718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.788719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.788786 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.789060 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.789062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.789063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.789211 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:27.789283 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:27.789667 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 06:11:27.789748 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:11:27.790536 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.779754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:34617 (local address 127.14.209.131:37691)
0620 06:11:27.780350 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:27.780388 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:27.781083 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:27.781518 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:27.781523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:27.781825 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:27.782247 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:27.782259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.783000 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.783002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:27.783899 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:27.783907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.784355 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.784362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:27.784586 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.785103 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:27.785123 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:27.786902 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:27.788382 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:27.788387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:27.788396 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:27.788634 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:27.788859 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:27.788861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:27.788862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:27.788983 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:27.789290 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:27.789294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:27.789486 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 06:11:27.789623 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 06:11:27.789822 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":110,"threads_started":1}
I20260620 06:11:27.791558 15685 heartbeater.cc:347] Connected to a master server at 127.14.209.190:34617
I20260620 06:11:27.791740 15685 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:27.792097 15685 heartbeater.cc:511] Master 127.14.209.190:34617 requested a full tablet report, sending...
I20260620 06:11:27.792953 15226 ts_manager.cc:194] Registered new tserver with Master: 2c29575c0126446da5afdd0bb72638e6 (127.14.209.131:41493)
I20260620 06:11:27.793470 15226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:37691
I20260620 06:11:27.798691 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:27.823727 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.814427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.14.209.129:40657)
0620 06:11:27.814832 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:27.814842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:27.814869 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:11:27.814933 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:27.814937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:27.815203 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:27.815325 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:27.815968 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.816824 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.817627 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.817954 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.818043 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.820305 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:27.820312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:27.820323 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:27.820346 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:27.821940 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.822392 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.822396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.822397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.822471 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.822709 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.822712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.822713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.822922 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:27.823192 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:27.823386 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 06:11:27.823543 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":156,"threads_started":1}
W20260620 06:11:27.824620 15331 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48466
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:27.848567 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:27.840724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.14.209.129:40657)
0620 06:11:27.840864 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:27.840867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:27.840905 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:11:27.841124 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:27.841127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:27.841175 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:27.841242 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:27.841816 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.842316 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.843098 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:27.843325 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:27.843400 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:27.845588 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:27.845591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:27.845593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:27.845614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:27.847042 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.847432 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.847435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.847436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.847512 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:27.847859 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:27.847863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:27.847864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:27.848084 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:27.848180 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:27.848375 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 06:11:27.848458 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:11:27.849579 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15288
I20260620 06:11:27.855029 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15422
I20260620 06:11:27.860466 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15556
I20260620 06:11:27.865455 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15194
2026-06-20T06:11:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2146 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15699](info): set up 2 sockets
Jun 20 06:11:27 dist-test-slave-zt01 krb5kdc[15699](info): commencing operation
krb5kdc: starting...
W20260620 06:11:28.902527 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:28 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:29.018814 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:36495
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:36495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:29.123587 15712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:29.123817 15712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:29.123855 15712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:29.127795 15712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:29.127847 15712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:29.127866 15712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:29.127883 15712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:29.127898 15712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:29.133028 15712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:36495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:36495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:29.134034 15712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:29.135020 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:29.143206 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.143208 15717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:29.143540 15712 server_base.cc:1034] running on GCE node
W20260620 06:11:29.143225 15718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:29.144296 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:29.145328 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:29.146508 15712 hybrid_clock.cc:648] HybridClock initialized: now 1781935889146476 us; error 44 us; skew 500 ppm
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:29.149344 15712 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:29.150177 15712 webserver.cc:460] Webserver started at http://127.14.209.190:38007/ using document root <none> and password file <none>
I20260620 06:11:29.150666 15712 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:29.150724 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:29.150878 15712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:29.152832 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "173291efdf099a8a6583fdf1a0a4f4c7"
server_key_iv: "aa093ccf5cf4c3a951a76d1606f9cc3d"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.153343 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "173291efdf099a8a6583fdf1a0a4f4c7"
server_key_iv: "aa093ccf5cf4c3a951a76d1606f9cc3d"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.156881 15712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:29.159035 15727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.160302 15712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:29.160451 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "173291efdf099a8a6583fdf1a0a4f4c7"
server_key_iv: "aa093ccf5cf4c3a951a76d1606f9cc3d"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.160552 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:29.181053 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:29.184530 15712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:29.184719 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:29.192260 15712 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:36495
I20260620 06:11:29.192315 15779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:36495 every 8 connection(s)
I20260620 06:11:29.193217 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:29.194505 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15712
I20260620 06:11:29.194635 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:29.194897 15174 external_mini_cluster.cc:1428] Setting key 3d18bbc5f523b0a04fa9d7db8a8edeed
I20260620 06:11:29.196230 15780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:29.202606 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3: Bootstrap starting.
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:29.204911 15780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:29.205842 15780 log.cc:824] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:29.207901 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3: No bootstrap required, opened a new log
I20260620 06:11:29.209940 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.196249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.14.209.190:36495)
0620 06:11:29.196681 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.196698 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.196739 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:11:29.197532 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.197553 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.197945 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.198281 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:29.198907 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.200406 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.201267 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.201968 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.202127 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.204580 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:29.204586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:29.204602 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:29.204631 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:29.206550 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.206973 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.206978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.206984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.207131 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.207450 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.207453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.207455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.207925 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:29.208105 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.208385 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.208659 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
I20260620 06:11:29.211962 15780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36495 } }
I20260620 06:11:29.212218 15780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:29.212323 15780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab12e35ae1342fcb0f2b13f0c3ad0e3, State: Initialized, Role: FOLLOWER
I20260620 06:11:29.212984 15780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36495 } }
I20260620 06:11:29.213122 15780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:29.213162 15780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:29.213236 15780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:29.214728 15780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36495 } }
I20260620 06:11:29.215168 15780 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab12e35ae1342fcb0f2b13f0c3ad0e3; no voters:
I20260620 06:11:29.215524 15780 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:29.215648 15785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:29.215879 15785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8ab12e35ae1342fcb0f2b13f0c3ad0e3, State: Running, Role: LEADER
I20260620 06:11:29.216207 15785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36495 } }
I20260620 06:11:29.216712 15780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:29.218498 15786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36495 } } }
I20260620 06:11:29.218607 15786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:29.218472 15787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab12e35ae1342fcb0f2b13f0c3ad0e3. Latest consensus state: current_term: 1 leader_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab12e35ae1342fcb0f2b13f0c3ad0e3" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36495 } } }
I20260620 06:11:29.218838 15787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:29.219264 15800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:29.221935 15800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:29.227324 15800 catalog_manager.cc:1347] Generated new cluster ID: fd3106695c0642b18b2e674a29167dc4
I20260620 06:11:29.227399 15800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:29.244068 15800 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:29.245085 15800 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:29.249486 15800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ab12e35ae1342fcb0f2b13f0c3ad0e3: Generated new TSK 0
I20260620 06:11:29.250157 15800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:29.286324 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36495
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:29.392951 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:29.393181 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:29.393224 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:29.397056 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:29.397099 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:29.397156 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:29.402115 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:29.402942 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:29.403820 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:29.412719 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.412717 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.412747 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:29.413053 15806 server_base.cc:1034] running on GCE node
I20260620 06:11:29.413416 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:29.414018 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:29.415145 15806 hybrid_clock.cc:648] HybridClock initialized: now 1781935889415109 us; error 38 us; skew 500 ppm
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:29.418076 15806 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:29.418970 15806 webserver.cc:460] Webserver started at http://127.14.209.129:35101/ using document root <none> and password file <none>
I20260620 06:11:29.419538 15806 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:29.419637 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:29.419857 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:29.421689 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "f26bb6aba0a3771ae04b73e9f9e074ba"
server_key_iv: "5dd22b05fddec52591c35f50e30bc3ec"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.422212 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "f26bb6aba0a3771ae04b73e9f9e074ba"
server_key_iv: "5dd22b05fddec52591c35f50e30bc3ec"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.425653 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:11:29.428058 15821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.429183 15806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:29.429314 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "f26bb6aba0a3771ae04b73e9f9e074ba"
server_key_iv: "5dd22b05fddec52591c35f50e30bc3ec"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.429427 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:29.461645 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:29.464869 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:29.465052 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:29.465590 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:29.466361 15806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:29.466432 15806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.466501 15806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:29.466554 15806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.475242 15806 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:39767
I20260620 06:11:29.475268 15934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:39767 every 8 connection(s)
I20260620 06:11:29.476048 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:29.482089 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15806
I20260620 06:11:29.482306 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:29.482537 15174 external_mini_cluster.cc:1428] Setting key d8419c818a895d30ca6159c3d3ca5e90
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:29.489692 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.477781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:43087 (local address 127.14.209.190:36495)
0620 06:11:29.477929 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.477934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.478806 (+ 872us) server_negotiation.cc:407] Connection header received
0620 06:11:29.479691 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.479693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.479738 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.479824 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:29.481004 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.481702 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.482902 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.483227 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.483331 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.485820 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:29.485823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:29.485826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:29.485849 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:29.487349 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.488001 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.488004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.488005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.488077 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.488537 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.488540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.488541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.488720 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:29.488818 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.489474 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.489579 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:11:29.490932 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.478219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:36495 (local address 127.14.209.129:43087)
0620 06:11:29.478651 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:29.478690 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:29.479485 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:29.479992 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:29.479999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:29.480327 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:29.480764 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.480777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.481830 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.481833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.482782 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.482791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.483399 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.483403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.483596 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.484260 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:29.484275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:29.485664 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:29.487501 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:29.487509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:29.487526 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:29.487894 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:29.488185 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:29.488191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:29.488196 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:29.488414 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:29.488851 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:29.488861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:29.489197 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 06:11:29.489433 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 06:11:29.489794 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:29.492271 15935 heartbeater.cc:347] Connected to a master server at 127.14.209.190:36495
I20260620 06:11:29.492467 15935 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:29.492930 15935 heartbeater.cc:511] Master 127.14.209.190:36495 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:11:29.494472 15744 ts_manager.cc:194] Registered new tserver with Master: f6126ca15b6d4dbfb5eecd970eeacdd7 (127.14.209.129:39767)
I20260620 06:11:29.495625 15744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:43087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:29.509450 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36495
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:29.613368 15940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:29.613611 15940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:29.613682 15940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:29.617708 15940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:29.617775 15940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:29.617901 15940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:29.622922 15940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:29.623792 15940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:29.624675 15940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:29.633319 15945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.633291 15948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.633318 15946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:29.633618 15940 server_base.cc:1034] running on GCE node
I20260620 06:11:29.633968 15940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:29.634507 15940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:29.635671 15940 hybrid_clock.cc:648] HybridClock initialized: now 1781935889635660 us; error 25 us; skew 500 ppm
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:29.638276 15940 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:29.639176 15940 webserver.cc:460] Webserver started at http://127.14.209.130:36683/ using document root <none> and password file <none>
I20260620 06:11:29.639699 15940 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:29.639789 15940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:29.639993 15940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:29.641813 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "aaaf961344274b85966892f6e54b2552"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "df11dc5aeb715ffa262c63ffbec586e5"
server_key_iv: "ef0631bc28d8193e35076bed4bbc5586"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.642352 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "aaaf961344274b85966892f6e54b2552"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "df11dc5aeb715ffa262c63ffbec586e5"
server_key_iv: "ef0631bc28d8193e35076bed4bbc5586"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.645975 15940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:11:29.648259 15955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.649356 15940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:29.649487 15940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "aaaf961344274b85966892f6e54b2552"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "df11dc5aeb715ffa262c63ffbec586e5"
server_key_iv: "ef0631bc28d8193e35076bed4bbc5586"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.649588 15940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:29.669615 15940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:29.672431 15940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:29.672619 15940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:29.673151 15940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:29.673911 15940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:29.673980 15940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.674048 15940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:29.674099 15940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.682709 15940 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:35419
I20260620 06:11:29.682753 16068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:35419 every 8 connection(s)
I20260620 06:11:29.683606 15940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:29.684950 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 15940
I20260620 06:11:29.685070 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:29.685353 15174 external_mini_cluster.cc:1428] Setting key f53bf670c15b75d00c0649d594efaccf
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:11:29.696934 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.685392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:59177 (local address 127.14.209.190:36495)
0620 06:11:29.685536 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.685541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.686526 (+ 985us) server_negotiation.cc:407] Connection header received
0620 06:11:29.687531 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.687534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.687581 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.687662 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:29.688866 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.689385 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.690580 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.690810 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.690919 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.693337 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:29.693340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:29.693342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:29.693363 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:29.694948 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.695483 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.695487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.695489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.695569 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.695917 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.695920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.695922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.696092 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:29.696199 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.696663 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.696778 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:11:29.697880 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.685810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:36495 (local address 127.14.209.130:59177)
0620 06:11:29.686366 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:29.686415 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:29.687346 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:29.687818 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:29.687825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:29.688289 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:29.688728 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.688741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.689517 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.689521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.690462 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.690472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.690958 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.690963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.691193 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.691778 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:29.691793 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:29.693167 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:29.695100 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:29.695106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:29.695119 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:29.695375 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:29.695668 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:29.695671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:29.695673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:29.695807 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:29.696197 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:29.696204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:29.696434 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 06:11:29.696605 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 06:11:29.696959 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260620 06:11:29.699090 16069 heartbeater.cc:347] Connected to a master server at 127.14.209.190:36495
I20260620 06:11:29.699338 16069 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:29.699927 16069 heartbeater.cc:511] Master 127.14.209.190:36495 requested a full tablet report, sending...
I20260620 06:11:29.700938 15744 ts_manager.cc:194] Registered new tserver with Master: aaaf961344274b85966892f6e54b2552 (127.14.209.130:35419)
I20260620 06:11:29.701884 15744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:59177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:29.714812 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36495
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:29.821313 16074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:29.821549 16074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:29.821626 16074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:29.825673 16074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:29.825722 16074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:29.825779 16074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:29.830906 16074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:29.831818 16074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:29.832769 16074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:29.842165 16082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.842032 16080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:29.842108 16079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:29.842509 16074 server_base.cc:1034] running on GCE node
I20260620 06:11:29.842845 16074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:29.843443 16074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:29.844604 16074 hybrid_clock.cc:648] HybridClock initialized: now 1781935889844590 us; error 25 us; skew 500 ppm
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:29.847426 16074 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:29.848287 16074 webserver.cc:460] Webserver started at http://127.14.209.131:44835/ using document root <none> and password file <none>
I20260620 06:11:29.848836 16074 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:29.848924 16074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:29.849138 16074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:29.851014 16074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "b5b31ba5e8324792bd0cd0599539b2f2"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "6b5ad8fa06bf7fdf6ceb41c126609fea"
server_key_iv: "cb32819daf7b26df7fc476feb3c95418"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.851608 16074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "b5b31ba5e8324792bd0cd0599539b2f2"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "6b5ad8fa06bf7fdf6ceb41c126609fea"
server_key_iv: "cb32819daf7b26df7fc476feb3c95418"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.855264 16074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:29.857522 16089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.858774 16074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:29.858930 16074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "b5b31ba5e8324792bd0cd0599539b2f2"
format_stamp: "Formatted at 2026-06-20 06:11:29 on dist-test-slave-zt01"
server_key: "6b5ad8fa06bf7fdf6ceb41c126609fea"
server_key_iv: "cb32819daf7b26df7fc476feb3c95418"
server_key_version: "encryptionkey@0"
I20260620 06:11:29.859038 16074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:29.872476 16074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:29.875433 16074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:29.875620 16074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:29.876165 16074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:29.876930 16074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:29.876999 16074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.877064 16074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:29.877106 16074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:29.885941 16074 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:43973
I20260620 06:11:29.885972 16202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:43973 every 8 connection(s)
I20260620 06:11:29.886747 16074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:29.890564 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16074
I20260620 06:11:29.890698 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:29.890978 15174 external_mini_cluster.cc:1428] Setting key 4170f2d02c9555f546c16beb0c4ab5c0
Jun 20 06:11:29 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:29.899278 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.888449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:40999 (local address 127.14.209.190:36495)
0620 06:11:29.888601 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.888605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.889339 (+ 734us) server_negotiation.cc:407] Connection header received
0620 06:11:29.890217 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.890220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.890275 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.890360 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:29.891591 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.892112 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.893204 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.893426 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.893535 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.895969 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:29.895977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:29.895980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:29.896000 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:29.897326 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.897841 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.897845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.897847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.897926 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.898245 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.898250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.898251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.898431 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:29.898542 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.898985 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.899118 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:11:29.899967 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.888783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:36495 (local address 127.14.209.131:40999)
0620 06:11:29.889202 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:29.889239 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:29.890024 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:29.890543 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:29.890549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:29.890956 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:29.891452 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.891465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.892233 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.892237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.893082 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.893090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.893569 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.893574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.893736 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.894301 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:29.894316 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:29.895807 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:29.897470 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:29.897475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:29.897488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:29.897734 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:29.898023 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:29.898025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:29.898027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:29.898152 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:29.898538 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:29.898544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:29.898761 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 06:11:29.898943 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 06:11:29.899244 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260620 06:11:29.901017 16203 heartbeater.cc:347] Connected to a master server at 127.14.209.190:36495
I20260620 06:11:29.901216 16203 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:29.901643 16203 heartbeater.cc:511] Master 127.14.209.190:36495 requested a full tablet report, sending...
I20260620 06:11:29.902577 15744 ts_manager.cc:194] Registered new tserver with Master: b5b31ba5e8324792bd0cd0599539b2f2 (127.14.209.131:43973)
I20260620 06:11:29.903182 15744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:40999
I20260620 06:11:29.904484 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:11:29.915575 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.907937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.14.209.190:36495)
0620 06:11:29.908046 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.908050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.908094 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:11:29.908355 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.908359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.908428 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.908546 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:29.909291 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.909801 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.910664 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.910936 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.911018 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.912393 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:29.912396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:29.912398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:29.912417 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:29.913835 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.914255 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.914258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.914259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.914337 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:29.914626 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:29.914629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:29.914630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:29.914896 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:29.915014 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.915388 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.915473 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:29.920081 15744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:29.922933 15744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:29.936496 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.930519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:43973 (local address 127.0.0.1:54538)
0620 06:11:29.930817 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:29.930862 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:29.930989 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:29.931936 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:29.931941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:29.931997 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:29.932127 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.932134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.933542 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.933545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.934889 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.934898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.935972 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.935975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.936086 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.936137 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 06:11:29.936235 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:11:29.936350 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"spinlock_wait_cycles":2432,"thread_start_us":55,"threads_started":1}
I20260620 06:11:29.936614 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.931543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.14.209.129:39767)
0620 06:11:29.931824 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.931837 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.932089 (+ 252us) server_negotiation.cc:407] Connection header received
0620 06:11:29.932166 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.932174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.932349 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.932501 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:29.932733 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.933808 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.935282 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.936034 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.936150 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.936253 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.936382 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.936466 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260620 06:11:29.936808 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.930648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.14.209.131:43973)
0620 06:11:29.931285 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.931299 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.931323 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:11:29.931394 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.931400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.931602 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.931764 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:29.932225 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.933404 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.934998 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.935838 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.935968 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.936095 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.936567 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.936665 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":120,"threads_started":1}
I20260620 06:11:29.936919 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.931397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39767 (local address 127.0.0.1:33008)
0620 06:11:29.931882 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:29.931895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:29.932007 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:29.932493 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:29.932496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:29.932515 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:29.932601 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.932606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.933942 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.933946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.935153 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.935162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.936168 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.936172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.936254 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.936286 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:11:29.936358 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:11:29.936434 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":85,"threads_started":1}
I20260620 06:11:29.937142 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.930519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:35419 (local address 127.0.0.1:42732)
0620 06:11:29.931246 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:29.931259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:29.931343 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:29.932836 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:29.932839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:29.932857 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:29.932972 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.932982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.934223 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.934226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.935361 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:29.935367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.936221 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.936228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:29.936332 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.936351 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:11:29.936399 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 06:11:29.936454 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":479,"thread_start_us":120,"threads_started":1}
I20260620 06:11:29.939287 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3c6507bf9249dc9f342cf8501c1e23 (DEFAULT_TABLE table=test-table [id=ac9840b0bf2a4dc68ec7c077a4b91d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:29.939302 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3c6507bf9249dc9f342cf8501c1e23 (DEFAULT_TABLE table=test-table [id=ac9840b0bf2a4dc68ec7c077a4b91d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:29.940196 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3c6507bf9249dc9f342cf8501c1e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:29.940203 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3c6507bf9249dc9f342cf8501c1e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:29.943717 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:29.930742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.14.209.130:35419)
0620 06:11:29.932386 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:29.932400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:29.932434 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:29.932503 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:29.932507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:29.932691 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:29.932847 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:29.933100 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.934068 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.935469 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:29.936080 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:29.937570 (+ 1490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:29.943439 (+ 5869us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:29.943491 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 06:11:29.943580 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":125,"threads_started":1}
I20260620 06:11:29.946220 16220 tablet_bootstrap.cc:492] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: Bootstrap starting.
I20260620 06:11:29.946336 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3c6507bf9249dc9f342cf8501c1e23 (DEFAULT_TABLE table=test-table [id=ac9840b0bf2a4dc68ec7c077a4b91d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:29.947340 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3c6507bf9249dc9f342cf8501c1e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:29.947788 16221 tablet_bootstrap.cc:492] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: Bootstrap starting.
I20260620 06:11:29.948664 16220 tablet_bootstrap.cc:654] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:29.949659 16220 log.cc:824] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:29.949703 16221 tablet_bootstrap.cc:654] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:29.950809 16221 log.cc:824] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:29.951517 16220 tablet_bootstrap.cc:492] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: No bootstrap required, opened a new log
I20260620 06:11:29.951730 16220 ts_tablet_manager.cc:1397] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:11:29.952689 16221 tablet_bootstrap.cc:492] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: No bootstrap required, opened a new log
I20260620 06:11:29.952867 16221 ts_tablet_manager.cc:1397] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:11:29.954067 16224 tablet_bootstrap.cc:492] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: Bootstrap starting.
I20260620 06:11:29.955412 16220 raft_consensus.cc:348] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:29.955677 16220 raft_consensus.cc:374] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:29.955740 16220 raft_consensus.cc:729] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b31ba5e8324792bd0cd0599539b2f2, State: Initialized, Role: FOLLOWER
I20260620 06:11:29.956159 16221 raft_consensus.cc:348] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:29.956183 16220 consensus_queue.cc:260] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:29.956357 16221 raft_consensus.cc:374] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:29.956419 16221 raft_consensus.cc:729] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6126ca15b6d4dbfb5eecd970eeacdd7, State: Initialized, Role: FOLLOWER
I20260620 06:11:29.956523 16224 tablet_bootstrap.cc:654] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:29.956898 16203 heartbeater.cc:503] Master 127.14.209.190:36495 was elected leader, sending a full tablet report...
I20260620 06:11:29.956859 16221 consensus_queue.cc:260] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:29.957248 16220 ts_tablet_manager.cc:1428] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:11:29.957698 16221 ts_tablet_manager.cc:1428] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:11:29.958030 15935 heartbeater.cc:503] Master 127.14.209.190:36495 was elected leader, sending a full tablet report...
I20260620 06:11:29.958015 16224 log.cc:824] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:29.959995 16224 tablet_bootstrap.cc:492] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: No bootstrap required, opened a new log
I20260620 06:11:29.960196 16224 ts_tablet_manager.cc:1397] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:11:29.963301 16224 raft_consensus.cc:348] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:29.963495 16224 raft_consensus.cc:374] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:29.963526 16224 raft_consensus.cc:729] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaf961344274b85966892f6e54b2552, State: Initialized, Role: FOLLOWER
I20260620 06:11:29.963912 16224 consensus_queue.cc:260] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:29.964527 16069 heartbeater.cc:503] Master 127.14.209.190:36495 was elected leader, sending a full tablet report...
I20260620 06:11:29.964818 16224 ts_tablet_manager.cc:1428] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 06:11:29.977154 15936 tablet.cc:2369] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:30.088568 16228 raft_consensus.cc:482] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:30.088828 16228 raft_consensus.cc:504] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:30.090009 16228 leader_election.cc:288] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b31ba5e8324792bd0cd0599539b2f2 (127.14.209.131:43973), f6126ca15b6d4dbfb5eecd970eeacdd7 (127.14.209.129:39767)
I20260620 06:11:30.094295 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.090313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39767 (local address 127.14.209.130:35483)
0620 06:11:30.090428 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:30.090447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:30.090538 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:30.090905 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:30.090908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:30.090948 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:30.091121 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:30.091128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.092072 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.092075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:30.092925 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:30.092934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.093763 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.093767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:30.093915 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.093943 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:11:30.094045 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 06:11:30.094136 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":768}
I20260620 06:11:30.094550 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.090476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:35483 (local address 127.14.209.129:39767)
0620 06:11:30.090619 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:30.090624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:30.090642 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:11:30.090705 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:30.090708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:30.090765 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:30.090882 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:30.091314 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.091949 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.093148 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.093635 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.093724 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.093763 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:30.094332 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 06:11:30.094415 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:11:30.094846 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.090313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:43973 (local address 127.14.209.130:34061)
0620 06:11:30.090855 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:30.090872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:30.090973 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:30.091302 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:30.091305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:30.091323 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:30.091459 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:30.091466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.092315 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.092317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:30.093293 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:30.093303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.094338 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.094342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:30.094438 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.094460 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:11:30.094609 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:11:30.094678 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":114,"thread_start_us":118,"threads_started":1}
I20260620 06:11:30.094913 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.090477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:34061 (local address 127.14.209.131:43973)
0620 06:11:30.090619 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:30.090624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:30.090965 (+ 341us) server_negotiation.cc:407] Connection header received
0620 06:11:30.091110 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:30.091113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:30.091168 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:30.091247 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:30.091626 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.092192 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.093434 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.094068 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.094182 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.094231 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:30.094665 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 06:11:30.094749 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:11:30.095198 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3c6507bf9249dc9f342cf8501c1e23" candidate_uuid: "aaaf961344274b85966892f6e54b2552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" is_pre_election: true
I20260620 06:11:30.095525 15889 raft_consensus.cc:2454] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaf961344274b85966892f6e54b2552 in term 0.
I20260620 06:11:30.095492 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3c6507bf9249dc9f342cf8501c1e23" candidate_uuid: "aaaf961344274b85966892f6e54b2552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" is_pre_election: true
I20260620 06:11:30.095737 16157 raft_consensus.cc:2454] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaf961344274b85966892f6e54b2552 in term 0.
I20260620 06:11:30.096020 15957 leader_election.cc:302] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaf961344274b85966892f6e54b2552, f6126ca15b6d4dbfb5eecd970eeacdd7; no voters:
I20260620 06:11:30.096431 16228 raft_consensus.cc:2781] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:30.096546 16228 raft_consensus.cc:482] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:30.096616 16228 raft_consensus.cc:3037] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:30.097708 16228 raft_consensus.cc:504] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:30.098083 16228 leader_election.cc:288] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [CANDIDATE]: Term 1 election: Requested vote from peers b5b31ba5e8324792bd0cd0599539b2f2 (127.14.209.131:43973), f6126ca15b6d4dbfb5eecd970eeacdd7 (127.14.209.129:39767)
I20260620 06:11:30.098546 15889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3c6507bf9249dc9f342cf8501c1e23" candidate_uuid: "aaaf961344274b85966892f6e54b2552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7"
I20260620 06:11:30.098536 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3c6507bf9249dc9f342cf8501c1e23" candidate_uuid: "aaaf961344274b85966892f6e54b2552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b31ba5e8324792bd0cd0599539b2f2"
I20260620 06:11:30.098662 16157 raft_consensus.cc:3037] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:30.098662 15889 raft_consensus.cc:3037] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:30.099929 15889 raft_consensus.cc:2454] T 7b3c6507bf9249dc9f342cf8501c1e23 P f6126ca15b6d4dbfb5eecd970eeacdd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaf961344274b85966892f6e54b2552 in term 1.
I20260620 06:11:30.099924 16157 raft_consensus.cc:2454] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaf961344274b85966892f6e54b2552 in term 1.
I20260620 06:11:30.100579 15957 leader_election.cc:302] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaf961344274b85966892f6e54b2552, f6126ca15b6d4dbfb5eecd970eeacdd7; no voters:
I20260620 06:11:30.100989 16228 raft_consensus.cc:2781] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:30.101377 16228 raft_consensus.cc:686] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [term 1 LEADER]: Becoming Leader. State: Replica: aaaf961344274b85966892f6e54b2552, State: Running, Role: LEADER
I20260620 06:11:30.101773 16228 consensus_queue.cc:237] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } }
I20260620 06:11:30.105679 15744 catalog_manager.cc:5495] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaaf961344274b85966892f6e54b2552 (127.14.209.130). New cstate: current_term: 1 leader_uuid: "aaaf961344274b85966892f6e54b2552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b31ba5e8324792bd0cd0599539b2f2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaaf961344274b85966892f6e54b2552" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 35419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6126ca15b6d4dbfb5eecd970eeacdd7" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39767 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:30.137728 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.126386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.14.209.129:39767)
0620 06:11:30.126632 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:30.126637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:30.127434 (+ 797us) server_negotiation.cc:407] Connection header received
0620 06:11:30.127522 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:30.127527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:30.127741 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:30.127874 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:30.128852 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.129539 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.130837 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.131164 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.131271 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.134152 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:30.134165 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:30.134180 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:30.134206 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:30.135957 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:30.136410 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:30.136415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:30.136416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:30.136501 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:30.136781 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:30.136784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:30.136786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:30.137034 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:30.137254 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:30.137412 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 06:11:30.137561 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260620 06:11:30.137972 16204 tablet.cc:2369] T 7b3c6507bf9249dc9f342cf8501c1e23 P b5b31ba5e8324792bd0cd0599539b2f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:30.165411 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.156267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.14.209.129:39767)
0620 06:11:30.156419 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:30.156423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:30.156456 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:30.156579 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:30.156582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:30.156634 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:30.156703 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:30.157400 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.158097 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.158938 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.159327 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.159438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.161907 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:30.161910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:30.161912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:30.161933 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:30.163682 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:30.164133 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:30.164135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:30.164137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:30.164210 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:30.164545 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:30.164548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:30.164549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:30.164716 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:30.164816 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:30.165103 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 06:11:30.165195 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 06:11:30.185218 16070 tablet.cc:2369] T 7b3c6507bf9249dc9f342cf8501c1e23 P aaaf961344274b85966892f6e54b2552: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:30.192744 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:30.183942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.14.209.129:39767)
0620 06:11:30.184102 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:30.184108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:30.184264 (+ 156us) server_negotiation.cc:407] Connection header received
0620 06:11:30.184434 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:30.184438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:30.184489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:30.184577 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:30.185230 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.185785 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.186618 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:30.186842 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:30.186987 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:30.189583 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:30.189586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:30.189589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:30.189619 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:30.191000 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:30.191450 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:30.191454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:30.191456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:30.191543 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:30.191887 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:30.191890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:30.191892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:30.192085 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:30.192214 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:30.192477 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 06:11:30.192590 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 06:11:30.194118 15849 tablet_service.cc:2908] Rejecting scan request for tablet 7b3c6507bf9249dc9f342cf8501c1e23: Uninitialized: clean time has not yet been initialized
I20260620 06:11:30.196204 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15806
I20260620 06:11:30.202527 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15940
I20260620 06:11:30.208668 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16074
I20260620 06:11:30.214300 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 15712
2026-06-20T06:11:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2348 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[16244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[16244](info): set up 2 sockets
Jun 20 06:11:30 dist-test-slave-zt01 krb5kdc[16244](info): commencing operation
krb5kdc: starting...
W20260620 06:11:31.254688 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:31 dist-test-slave-zt01 krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:31.373418 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:42785
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:42785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:31.477262 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:31.477551 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:31.477622 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:31.481535 16257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:31.481628 16257 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:11:31.481660 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:31.481701 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:31.481742 16257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:31.481778 16257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:31.486901 16257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:42785
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:31.487892 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:31.488878 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:31.497195 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:31.497361 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:31.497231 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:31.497262 16257 server_base.cc:1034] running on GCE node
I20260620 06:11:31.498040 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:31.499095 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:31.500283 16257 hybrid_clock.cc:648] HybridClock initialized: now 1781935891500266 us; error 30 us; skew 500 ppm
Jun 20 06:11:31 dist-test-slave-zt01 krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:31.507292 16257 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:31.508229 16257 webserver.cc:460] Webserver started at http://127.14.209.190:40661/ using document root <none> and password file <none>
I20260620 06:11:31.508781 16257 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:31.508841 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:31.509106 16257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:31.511034 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "d43da5d67bfc4762af37363585949a87"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "71fe86dc6d1ff5d2d45c9bd2009d7191"
server_key_iv: "161f68e7511ae9bc277e958112080e05"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.511662 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "d43da5d67bfc4762af37363585949a87"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "71fe86dc6d1ff5d2d45c9bd2009d7191"
server_key_iv: "161f68e7511ae9bc277e958112080e05"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.515224 16257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:11:31.517515 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:31.518720 16257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:31.518814 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "d43da5d67bfc4762af37363585949a87"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "71fe86dc6d1ff5d2d45c9bd2009d7191"
server_key_iv: "161f68e7511ae9bc277e958112080e05"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.518888 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:31.544170 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:31.547250 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:31.547391 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:31.555047 16257 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:42785
I20260620 06:11:31.555244 16334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:42785 every 8 connection(s)
I20260620 06:11:31.556008 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:31.558493 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16257
I20260620 06:11:31.558586 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:31.558728 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:31.558890 15174 external_mini_cluster.cc:1428] Setting key 5bd4acf64735dff8fe76b1f82ab75bbb
I20260620 06:11:31.565883 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: Bootstrap starting.
Jun 20 06:11:31 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:31.568037 16335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:31.568797 16335 log.cc:824] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:31.570578 16335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: No bootstrap required, opened a new log
I20260620 06:11:31.573348 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:31.560253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.14.209.190:42785)
0620 06:11:31.560656 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:31.560673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:31.560712 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:11:31.561665 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:31.561687 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:31.562089 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:31.562440 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:31.563075 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.564524 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.565376 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.566037 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.566185 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:31.568570 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:31.568576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:31.568593 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:31.568625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:31.570798 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:31.571296 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:31.571300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:31.571304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:31.571413 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:31.571694 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:31.571695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:31.571697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:31.572048 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:31.572183 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:31.572443 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 06:11:31.572696 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260620 06:11:31.573925 16335 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43da5d67bfc4762af37363585949a87" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42785 } }
I20260620 06:11:31.574138 16335 raft_consensus.cc:374] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:31.574227 16335 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43da5d67bfc4762af37363585949a87, State: Initialized, Role: FOLLOWER
I20260620 06:11:31.574678 16335 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43da5d67bfc4762af37363585949a87" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42785 } }
I20260620 06:11:31.574841 16335 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:31.574924 16335 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:31.575019 16335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:31.575877 16335 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43da5d67bfc4762af37363585949a87" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42785 } }
I20260620 06:11:31.576191 16335 leader_election.cc:302] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43da5d67bfc4762af37363585949a87; no voters:
I20260620 06:11:31.576535 16335 leader_election.cc:288] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:31.576668 16340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:31.576892 16340 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [term 1 LEADER]: Becoming Leader. State: Replica: d43da5d67bfc4762af37363585949a87, State: Running, Role: LEADER
I20260620 06:11:31.577214 16340 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43da5d67bfc4762af37363585949a87" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42785 } }
I20260620 06:11:31.577636 16335 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:31.578694 16341 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43da5d67bfc4762af37363585949a87. Latest consensus state: current_term: 1 leader_uuid: "d43da5d67bfc4762af37363585949a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43da5d67bfc4762af37363585949a87" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42785 } } }
I20260620 06:11:31.578820 16341 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:31.579200 16349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:31.579157 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d43da5d67bfc4762af37363585949a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43da5d67bfc4762af37363585949a87" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42785 } } }
I20260620 06:11:31.579391 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:31.581663 16349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:31.586483 16349 catalog_manager.cc:1347] Generated new cluster ID: c245c6f8e24b455292cad6c5b3956df2
I20260620 06:11:31.586544 16349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:31.597000 16349 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:31.598083 16349 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:31.608119 16349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: Generated new TSK 0
I20260620 06:11:31.608816 16349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:31.637526 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:42785
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:31.745517 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:31.745748 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:31.745791 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:31.749758 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:31.749811 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:31.749910 16361 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:11:31.749960 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:31.755234 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:31.756237 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:31.757195 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:31.766170 16366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:31.766279 16367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:31.766178 16369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:31.766389 16361 server_base.cc:1034] running on GCE node
I20260620 06:11:31.766834 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:31.767457 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:31.768623 16361 hybrid_clock.cc:648] HybridClock initialized: now 1781935891768587 us; error 49 us; skew 500 ppm
Jun 20 06:11:31 dist-test-slave-zt01 krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:31.771241 16361 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:31.771998 16361 webserver.cc:460] Webserver started at http://127.14.209.129:45853/ using document root <none> and password file <none>
I20260620 06:11:31.772511 16361 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:31.772594 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:31.772799 16361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:31.774647 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "a4f97e13a1a645e39b80a1083267b24d"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "c3b9219b0893735c09af593666f366be"
server_key_iv: "c46d7c79487de2a7f94939457b006a52"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.775279 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "a4f97e13a1a645e39b80a1083267b24d"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "c3b9219b0893735c09af593666f366be"
server_key_iv: "c46d7c79487de2a7f94939457b006a52"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.778831 16361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:11:31.781358 16376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:31.782502 16361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:31.782637 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "a4f97e13a1a645e39b80a1083267b24d"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "c3b9219b0893735c09af593666f366be"
server_key_iv: "c46d7c79487de2a7f94939457b006a52"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.782748 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:31.804036 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:31.807533 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:31.807842 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:31.809259 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:31.809336 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:31.809402 16361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:31.809449 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:11:31 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:31.819885 16361 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:46865
I20260620 06:11:31.819977 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:46865 every 8 connection(s)
I20260620 06:11:31.820710 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:31.823163 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16361
I20260620 06:11:31.823297 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:31.823572 15174 external_mini_cluster.cc:1428] Setting key e9930bb122b959762385731c4cd94c94
I20260620 06:11:31.823887 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:31.810793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:38755 (local address 127.14.209.190:42785)
0620 06:11:31.810969 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:31.810975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:31.811861 (+ 886us) server_negotiation.cc:407] Connection header received
0620 06:11:31.813143 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:31.813148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:31.813227 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:31.813333 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:31.814763 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.815523 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.816821 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.817118 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.817234 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:31.820063 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:31.820066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:31.820069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:31.820099 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:31.821790 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:31.822291 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:31.822295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:31.822297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:31.822379 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:31.822743 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:31.822748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:31.822750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:31.822952 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:31.823081 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:31.823596 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 06:11:31.823714 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:11:31.824680 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:31.811107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.14.209.129:38755)
0620 06:11:31.811655 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:31.811722 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:31.812921 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:31.813453 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:31.813460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:31.813928 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:31.814620 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:31.814642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.815650 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.815654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:31.816699 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:31.816709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.817272 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.817275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:31.817470 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:31.818071 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:31.818115 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:31.819853 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:31.821904 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:31.821910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:31.821930 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:31.822189 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:31.822475 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:31.822478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:31.822480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:31.822650 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:31.823074 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:31.823080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:31.823342 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 06:11:31.823541 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 06:11:31.823828 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":131,"threads_started":1}
I20260620 06:11:31.826051 16492 heartbeater.cc:347] Connected to a master server at 127.14.209.190:42785
I20260620 06:11:31.826179 16492 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:31.826722 16492 heartbeater.cc:511] Master 127.14.209.190:42785 requested a full tablet report, sending...
I20260620 06:11:31.828258 16288 ts_manager.cc:194] Registered new tserver with Master: a4f97e13a1a645e39b80a1083267b24d (127.14.209.129:46865)
I20260620 06:11:31.829599 16288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:38755
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:31.836390 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:31.828153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.14.209.129:45023)
0620 06:11:31.828468 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:31.828483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:31.828585 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:31.828826 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:31.828828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:31.829049 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:31.829153 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:31.829158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.830116 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.830122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:31.831180 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:31.831192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.831768 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.831771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:31.832004 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:31.832527 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:31.832541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:31.832864 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:31.834844 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:31.834846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:31.834848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:31.835037 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:31.835402 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:31.835405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:31.835406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:31.835484 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:31.835916 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:31.835919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:31.836040 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 06:11:31.836140 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:11:31.836242 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20260620 06:11:31.836400 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:31.828232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:45023 (local address 127.14.209.190:42785)
0620 06:11:31.828369 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:31.828373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:31.828561 (+ 188us) server_negotiation.cc:407] Connection header received
0620 06:11:31.828660 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:31.828663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:31.828699 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:31.828777 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:31.829264 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.829969 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.831316 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:31.831623 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:31.831731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:31.833008 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:31.833015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:31.833021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:31.833052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:31.834678 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:31.835205 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:31.835210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:31.835213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:31.835302 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:31.835575 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:31.835579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:31.835581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:31.835813 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:31.835930 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:31.836211 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 06:11:31.836296 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:31.855738 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:42785
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:31.960881 16501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:31.961127 16501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:31.961169 16501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:31.965045 16501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:31.965092 16501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:31.965129 16501 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:11:31.965159 16501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:31.970000 16501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:31.970928 16501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:31.971879 16501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:31.981138 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:31.981165 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:31.981144 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:31.981338 16501 server_base.cc:1034] running on GCE node
I20260620 06:11:31.982028 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:31.982556 16501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:31.983707 16501 hybrid_clock.cc:648] HybridClock initialized: now 1781935891983682 us; error 37 us; skew 500 ppm
Jun 20 06:11:31 dist-test-slave-zt01 krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:31.986310 16501 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:31.987238 16501 webserver.cc:460] Webserver started at http://127.14.209.130:35449/ using document root <none> and password file <none>
I20260620 06:11:31.987720 16501 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:31.987775 16501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:31.987921 16501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:31.989754 16501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "90d6009014754692ab10d1bd6ae353bb"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "95a43f35be593a18c602df8d7edf3fcf"
server_key_iv: "b0dc4edee55da20d2cf1174c74ea4ba7"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.990218 16501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "90d6009014754692ab10d1bd6ae353bb"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "95a43f35be593a18c602df8d7edf3fcf"
server_key_iv: "b0dc4edee55da20d2cf1174c74ea4ba7"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.993688 16501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:31.996201 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:31.997768 16501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:31.997933 16501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "90d6009014754692ab10d1bd6ae353bb"
format_stamp: "Formatted at 2026-06-20 06:11:31 on dist-test-slave-zt01"
server_key: "95a43f35be593a18c602df8d7edf3fcf"
server_key_iv: "b0dc4edee55da20d2cf1174c74ea4ba7"
server_key_version: "encryptionkey@0"
I20260620 06:11:31.998036 16501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:32.023340 16501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:32.026000 16501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:32.026193 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:32.027679 16501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:32.027725 16501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:32.027762 16501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:32.027815 16501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:32.038053 16501 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:46873
I20260620 06:11:32.038121 16631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:46873 every 8 connection(s)
I20260620 06:11:32.039039 16501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:32.041445 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16501
I20260620 06:11:32.041579 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:32.041649 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.028407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:34049 (local address 127.14.209.190:42785)
0620 06:11:32.028542 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.028547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.029380 (+ 833us) server_negotiation.cc:407] Connection header received
0620 06:11:32.030556 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.030562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.030625 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.030732 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.032332 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.033198 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.034487 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.034879 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.034982 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.037726 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.037730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.037732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.037759 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.039584 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.040164 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.040169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.040171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.040262 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.040593 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.040598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.040600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.040772 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.040878 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.041365 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.041482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:32.041916 15174 external_mini_cluster.cc:1428] Setting key bf8e151f94731032ec28f5a754f515e5
I20260620 06:11:32.042430 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.028799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.14.209.130:34049)
0620 06:11:32.029215 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.029261 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.030295 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.031018 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.031024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.031472 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:32.032162 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.032180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.033333 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.033336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.034369 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.034378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.035012 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.035016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.035212 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.035796 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:32.035816 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:32.037566 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:32.039699 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.039703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.039715 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.040051 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.040354 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.040359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.040361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.040498 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.040886 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:32.040895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:32.041124 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.041324 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.041615 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260620 06:11:32.043797 16632 heartbeater.cc:347] Connected to a master server at 127.14.209.190:42785
I20260620 06:11:32.043946 16632 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:32.044472 16632 heartbeater.cc:511] Master 127.14.209.190:42785 requested a full tablet report, sending...
I20260620 06:11:32.045531 16288 ts_manager.cc:194] Registered new tserver with Master: 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873)
I20260620 06:11:32.046165 16288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:34049
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:11:32.053779 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.045730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:54463 (local address 127.14.209.190:42785)
0620 06:11:32.045846 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.045851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.046536 (+ 685us) server_negotiation.cc:407] Connection header received
0620 06:11:32.046665 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.046671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.046723 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.046808 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.047306 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.048164 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.048976 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.049387 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.049519 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.050581 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.050586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.050591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.050621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.052220 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.052614 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.052619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.052621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.052703 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.052972 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.052977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.052978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.053175 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.053281 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.053539 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.053636 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:11:32.053789 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.045653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.14.209.130:54463)
0620 06:11:32.046444 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.046477 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.046572 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.046837 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.046840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.047075 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:32.047186 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.047193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.048311 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.048314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.048853 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.048860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.049536 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.049540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.049644 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.050121 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:32.050132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:32.050421 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:32.052323 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.052325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.052326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.052504 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.052803 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.052807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.052814 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.052883 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.053281 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:32.053284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:32.053367 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.053469 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.053600 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:32.071240 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:42785
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:32.181334 16641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:32.181561 16641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:32.181603 16641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:32.185487 16641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:32.185541 16641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:32.185580 16641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:11:32.185611 16641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:32.190769 16641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:32.191720 16641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:32.192601 16641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:32.201829 16649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:32.201841 16646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:32.201826 16647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:32.202208 16641 server_base.cc:1034] running on GCE node
I20260620 06:11:32.202605 16641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:32.203293 16641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:32.204486 16641 hybrid_clock.cc:648] HybridClock initialized: now 1781935892204469 us; error 36 us; skew 500 ppm
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935892, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:32.207674 16641 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:32.208544 16641 webserver.cc:460] Webserver started at http://127.14.209.131:36347/ using document root <none> and password file <none>
I20260620 06:11:32.209085 16641 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:32.209175 16641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:32.209396 16641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:32.211380 16641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "2ab6c020c3c84dc791e5655b2b094b80"
format_stamp: "Formatted at 2026-06-20 06:11:32 on dist-test-slave-zt01"
server_key: "1613003d43656cb7d0128f5a7dafbbe5"
server_key_iv: "7b9b4c786bcc95051a951fbcebaade29"
server_key_version: "encryptionkey@0"
I20260620 06:11:32.211912 16641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab6c020c3c84dc791e5655b2b094b80"
format_stamp: "Formatted at 2026-06-20 06:11:32 on dist-test-slave-zt01"
server_key: "1613003d43656cb7d0128f5a7dafbbe5"
server_key_iv: "7b9b4c786bcc95051a951fbcebaade29"
server_key_version: "encryptionkey@0"
I20260620 06:11:32.215448 16641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:32.217731 16656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:32.218950 16641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:32.219137 16641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "2ab6c020c3c84dc791e5655b2b094b80"
format_stamp: "Formatted at 2026-06-20 06:11:32 on dist-test-slave-zt01"
server_key: "1613003d43656cb7d0128f5a7dafbbe5"
server_key_iv: "7b9b4c786bcc95051a951fbcebaade29"
server_key_version: "encryptionkey@0"
I20260620 06:11:32.219254 16641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:32.243508 16641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:32.246886 16641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:32.247114 16641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:32.248492 16641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:32.248569 16641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:32.248642 16641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:32.248685 16641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935892, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:32.260118 16641 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:37347
I20260620 06:11:32.260193 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:37347 every 8 connection(s)
I20260620 06:11:32.261083 16641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:32.264379 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.250055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:42543 (local address 127.14.209.190:42785)
0620 06:11:32.250186 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.250190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.251261 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 06:11:32.252530 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.252533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.252587 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.252693 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.254659 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.255589 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.257494 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.257837 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.257954 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.260722 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.260731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.260733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.260755 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.262376 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.262873 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.262878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.262880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.262959 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.263320 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.263323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.263325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.263498 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.263604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.264082 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.264197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:32.265174 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.250478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.14.209.131:42543)
0620 06:11:32.251107 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.251167 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.252284 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.252863 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.252873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.253350 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:32.254392 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.254425 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.255773 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.255779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.257328 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.257344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.258090 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.258093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.258225 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.258822 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:32.258848 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:32.260549 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:32.262494 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.262500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.262513 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.262757 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.263075 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.263079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.263081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.263218 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.263611 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:32.263619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:32.263851 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.264023 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.264319 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":130,"threads_started":1}
I20260620 06:11:32.266371 16772 heartbeater.cc:347] Connected to a master server at 127.14.209.190:42785
I20260620 06:11:32.266532 16772 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:32.267032 16772 heartbeater.cc:511] Master 127.14.209.190:42785 requested a full tablet report, sending...
I20260620 06:11:32.267915 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16641
I20260620 06:11:32.268028 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:32.268066 16289 ts_manager.cc:194] Registered new tserver with Master: 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347)
I20260620 06:11:32.268298 15174 external_mini_cluster.cc:1428] Setting key 3c392a17694f469dfa38a570578591cf
I20260620 06:11:32.268847 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:42543
I20260620 06:11:32.270195 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:11:32.277208 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.268684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:53341 (local address 127.14.209.190:42785)
0620 06:11:32.268801 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.268806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.269174 (+ 368us) server_negotiation.cc:407] Connection header received
0620 06:11:32.269341 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.269347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.269402 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.269482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.269943 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.270724 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.271680 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.272060 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.272171 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.273434 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.273437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.273439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.273459 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.274868 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.275392 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.275398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.275399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.275486 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.275749 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.275756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.275759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.275999 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.276116 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.276945 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.277055 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:11:32.277251 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.268758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.14.209.131:53341)
0620 06:11:32.269098 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.269112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.269231 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.269511 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.269514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.269718 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:32.269820 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.269827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.270855 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.270858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.271569 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.271577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.272202 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.272208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.272333 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.272798 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:32.272814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:32.273301 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:32.274991 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.274995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.274997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.275284 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.275586 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.275588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.275589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.275659 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.276102 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:32.276104 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:32.276173 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.276681 (+ 508us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.277034 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935892, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935892, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:32.296378 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.288492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.14.209.190:42785)
0620 06:11:32.288607 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.288610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.288621 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:11:32.288650 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.288652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.288684 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.288748 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.289378 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.289857 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.290733 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.290937 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.291039 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.293363 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.293366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.293368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.293388 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.294855 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.295385 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.295388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.295389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.295465 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.295718 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.295720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.295721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.295868 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.295966 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.296195 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.296275 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:32.299144 16289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:32.303285 16289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:32.317842 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.312914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46865 (local address 127.0.0.1:46992)
0620 06:11:32.313294 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.313331 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.313422 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.314070 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.314074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.314093 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.314176 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.314181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.315261 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.315265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.316421 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.316430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.317349 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.317352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.317461 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.317503 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.317590 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.317697 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":68,"thread_start_us":93,"threads_started":1}
I20260620 06:11:32.317886 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.313330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.14.209.129:46865)
0620 06:11:32.313640 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.313652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.313673 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:11:32.313747 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.313753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.313921 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.314070 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.314291 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.315150 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.316543 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.317213 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.317470 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.317600 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.317652 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.317746 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260620 06:11:32.318111 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.312265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:37347 (local address 127.0.0.1:33788)
0620 06:11:32.312885 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.312899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.312967 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.314198 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.314201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.314223 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.314334 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.314343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.315643 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.315646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.316744 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.316755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.317725 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.317728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.317819 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.317839 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.317916 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.317987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":98,"threads_started":1}
I20260620 06:11:32.318463 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.312166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46873 (local address 127.0.0.1:39006)
0620 06:11:32.312609 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.312660 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.312784 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.313508 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.313514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.313537 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.313673 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.313680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.315005 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.315009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.316425 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.316435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.317320 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.317327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.318174 (+ 847us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.318202 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.318270 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.318348 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":69,"thread_start_us":65,"threads_started":1}
I20260620 06:11:32.318598 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.312301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.14.209.130:46873)
0620 06:11:32.312714 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.312725 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.312747 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:11:32.312976 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.312983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.313201 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.313347 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.313802 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.314887 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.316558 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.317193 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.317344 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.317473 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.318361 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.318450 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":100,"threads_started":1}
I20260620 06:11:32.320574 16426 tablet_service.cc:1467] Processing CreateTablet for tablet d003f6f1f8c649f8ba3a3d02fdf9795f (DEFAULT_TABLE table=test-table [id=397ac32fda294337a92e58fd9c005501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:32.320638 16566 tablet_service.cc:1467] Processing CreateTablet for tablet d003f6f1f8c649f8ba3a3d02fdf9795f (DEFAULT_TABLE table=test-table [id=397ac32fda294337a92e58fd9c005501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:32.321488 16566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d003f6f1f8c649f8ba3a3d02fdf9795f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.321489 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d003f6f1f8c649f8ba3a3d02fdf9795f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.328255 16795 tablet_bootstrap.cc:492] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Bootstrap starting.
I20260620 06:11:32.329247 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.312920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.14.209.131:37347)
0620 06:11:32.313704 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.313715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.313731 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:32.313790 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.313794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.313944 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.314070 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.314460 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.315526 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.316865 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.317607 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.317749 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.317849 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.318356 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.318441 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":108,"threads_started":1}
I20260620 06:11:32.329674 16796 tablet_bootstrap.cc:492] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Bootstrap starting.
I20260620 06:11:32.330524 16795 tablet_bootstrap.cc:654] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.331393 16795 log.cc:824] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:32.331676 16796 tablet_bootstrap.cc:654] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.331598 16706 tablet_service.cc:1467] Processing CreateTablet for tablet d003f6f1f8c649f8ba3a3d02fdf9795f (DEFAULT_TABLE table=test-table [id=397ac32fda294337a92e58fd9c005501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:32.332427 16796 log.cc:824] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:32.332643 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d003f6f1f8c649f8ba3a3d02fdf9795f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.333421 16795 tablet_bootstrap.cc:492] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: No bootstrap required, opened a new log
I20260620 06:11:32.333602 16795 ts_tablet_manager.cc:1397] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:11:32.334177 16796 tablet_bootstrap.cc:492] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: No bootstrap required, opened a new log
I20260620 06:11:32.334379 16796 ts_tablet_manager.cc:1397] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:11:32.337308 16796 raft_consensus.cc:348] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.337576 16796 raft_consensus.cc:374] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.337630 16796 raft_consensus.cc:729] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6009014754692ab10d1bd6ae353bb, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.337484 16795 raft_consensus.cc:348] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.337704 16795 raft_consensus.cc:374] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.337752 16795 raft_consensus.cc:729] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f97e13a1a645e39b80a1083267b24d, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.338099 16796 consensus_queue.cc:260] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.338179 16795 consensus_queue.cc:260] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.338677 16632 heartbeater.cc:503] Master 127.14.209.190:42785 was elected leader, sending a full tablet report...
I20260620 06:11:32.339027 16796 ts_tablet_manager.cc:1428] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260620 06:11:32.339082 16795 ts_tablet_manager.cc:1428] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 06:11:32.339376 16492 heartbeater.cc:503] Master 127.14.209.190:42785 was elected leader, sending a full tablet report...
I20260620 06:11:32.340930 16801 tablet_bootstrap.cc:492] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Bootstrap starting.
I20260620 06:11:32.343189 16801 tablet_bootstrap.cc:654] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.343843 16801 log.cc:824] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:32.345371 16801 tablet_bootstrap.cc:492] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: No bootstrap required, opened a new log
I20260620 06:11:32.345569 16801 ts_tablet_manager.cc:1397] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 06:11:32.348260 16801 raft_consensus.cc:348] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.348491 16801 raft_consensus.cc:374] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.348578 16801 raft_consensus.cc:729] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab6c020c3c84dc791e5655b2b094b80, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.348958 16801 consensus_queue.cc:260] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.349578 16772 heartbeater.cc:503] Master 127.14.209.190:42785 was elected leader, sending a full tablet report...
I20260620 06:11:32.349787 16801 ts_tablet_manager.cc:1428] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260620 06:11:32.464775 16800 raft_consensus.cc:482] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:32.464967 16800 raft_consensus.cc:504] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.466001 16800 leader_election.cc:288] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873), 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347)
I20260620 06:11:32.469820 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.466371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:35325 (local address 127.14.209.131:37347)
0620 06:11:32.466550 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.466557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.466579 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:11:32.466808 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.466812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.466861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.466945 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.467279 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.467843 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.468885 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.469277 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.469414 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.469462 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.469638 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.469701 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:11:32.469839 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.466216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:37347 (local address 127.14.209.129:35325)
0620 06:11:32.466500 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.466517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.466676 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.466992 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.466995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.467011 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.467167 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.467172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.467953 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.467958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.468751 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.468759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.469407 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.469410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.469506 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.469535 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.469623 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.469694 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260620 06:11:32.469992 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.466381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:52211 (local address 127.14.209.130:46873)
0620 06:11:32.466499 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.466505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.466936 (+ 431us) server_negotiation.cc:407] Connection header received
0620 06:11:32.467129 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.467132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.467170 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.467253 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.467498 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.468059 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.469032 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.469429 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.469552 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.469603 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.469775 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.469841 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:32.470224 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.466216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46873 (local address 127.14.209.129:52211)
0620 06:11:32.466833 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.466850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.466970 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.467261 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.467264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.467278 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.467388 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.467393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.468170 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.468175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.468903 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.468911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.469546 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.469550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.469658 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.469675 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.469762 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.469826 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":375,"thread_start_us":96,"threads_started":1}
I20260620 06:11:32.470552 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d003f6f1f8c649f8ba3a3d02fdf9795f" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab6c020c3c84dc791e5655b2b094b80" is_pre_election: true
I20260620 06:11:32.470855 16726 raft_consensus.cc:2454] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 0.
I20260620 06:11:32.470849 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d003f6f1f8c649f8ba3a3d02fdf9795f" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6009014754692ab10d1bd6ae353bb" is_pre_election: true
I20260620 06:11:32.471148 16586 raft_consensus.cc:2454] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 0.
I20260620 06:11:32.471314 16379 leader_election.cc:302] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab6c020c3c84dc791e5655b2b094b80, a4f97e13a1a645e39b80a1083267b24d; no voters:
I20260620 06:11:32.471587 16800 raft_consensus.cc:2781] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:32.471674 16800 raft_consensus.cc:482] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:32.471726 16800 raft_consensus.cc:3037] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.472736 16800 raft_consensus.cc:504] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.473122 16800 leader_election.cc:288] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 election: Requested vote from peers 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873), 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347)
I20260620 06:11:32.473415 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d003f6f1f8c649f8ba3a3d02fdf9795f" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6009014754692ab10d1bd6ae353bb"
I20260620 06:11:32.473484 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d003f6f1f8c649f8ba3a3d02fdf9795f" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab6c020c3c84dc791e5655b2b094b80"
I20260620 06:11:32.473527 16586 raft_consensus.cc:3037] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.473627 16726 raft_consensus.cc:3037] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.474543 16586 raft_consensus.cc:2454] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 1.
I20260620 06:11:32.474716 16726 raft_consensus.cc:2454] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 1.
I20260620 06:11:32.474865 16378 leader_election.cc:302] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d6009014754692ab10d1bd6ae353bb, a4f97e13a1a645e39b80a1083267b24d; no voters:
I20260620 06:11:32.475028 16800 raft_consensus.cc:2781] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:32.475279 16800 raft_consensus.cc:686] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 1 LEADER]: Becoming Leader. State: Replica: a4f97e13a1a645e39b80a1083267b24d, State: Running, Role: LEADER
I20260620 06:11:32.475592 16800 consensus_queue.cc:237] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } }
I20260620 06:11:32.478708 16287 catalog_manager.cc:5495] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d reported cstate change: term changed from 0 to 1, leader changed from <none> to a4f97e13a1a645e39b80a1083267b24d (127.14.209.129). New cstate: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } health_report { overall_health: HEALTHY } } }
I20260620 06:11:32.509418 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.505841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.14.209.129:46865)
0620 06:11:32.506002 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.506005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.506017 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:32.506078 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.506080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.506128 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.506212 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:32.506555 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.507046 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.507953 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.508171 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.508250 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.508609 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.508973 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.509070 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.509174 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.509220 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 06:11:32.512537 16773 tablet.cc:2369] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:32.514989 16586 raft_consensus.cc:1261] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Refusing update from remote peer a4f97e13a1a645e39b80a1083267b24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:32.514987 16726 raft_consensus.cc:1261] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Refusing update from remote peer a4f97e13a1a645e39b80a1083267b24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:32.515751 16806 consensus_queue.cc:1035] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:32.516010 16800 consensus_queue.cc:1035] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:32.525457 16810 mvcc.cc:205] Tried to move back new op lower bound from 7298809415734235136 to 7298809415582593024. Current Snapshot: MvccSnapshot[applied={T|T < 7298809415734235136 or (T in {7298809415734235136})}]
I20260620 06:11:32.526782 16811 mvcc.cc:205] Tried to move back new op lower bound from 7298809415734235136 to 7298809415582593024. Current Snapshot: MvccSnapshot[applied={T|T < 7298809415734235136}]
I20260620 06:11:32.534327 16287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:32.534497 16287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:32.536996 16287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: Sending DeleteTablet for 3 replicas of tablet d003f6f1f8c649f8ba3a3d02fdf9795f
I20260620 06:11:32.537804 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet d003f6f1f8c649f8ba3a3d02fdf9795f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:32 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:33788
I20260620 06:11:32.537935 16566 tablet_service.cc:1514] Processing DeleteTablet for tablet d003f6f1f8c649f8ba3a3d02fdf9795f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:32 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:39006
I20260620 06:11:32.538425 16820 tablet_replica.cc:330] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: stopping tablet replica
I20260620 06:11:32.538393 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet d003f6f1f8c649f8ba3a3d02fdf9795f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:32 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:46992
I20260620 06:11:32.538530 16289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:11:32.538793 16821 tablet_replica.cc:330] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: stopping tablet replica
W20260620 06:11:32.538935 16289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:32.539075 16820 raft_consensus.cc:2229] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:32.539088 16821 raft_consensus.cc:2229] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:32.539413 16821 raft_consensus.cc:2258] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:32.539533 16820 raft_consensus.cc:2258] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:32.540850 16821 ts_tablet_manager.cc:1905] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:32.540884 16820 ts_tablet_manager.cc:1905] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:32.543709 16821 ts_tablet_manager.cc:1918] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:32.543727 16820 ts_tablet_manager.cc:1918] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:32.543923 16820 log.cc:1196] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/d003f6f1f8c649f8ba3a3d02fdf9795f
I20260620 06:11:32.544315 16820 ts_tablet_manager.cc:1939] T d003f6f1f8c649f8ba3a3d02fdf9795f P 90d6009014754692ab10d1bd6ae353bb: Deleting consensus metadata
I20260620 06:11:32.544605 16706 tablet_service.cc:1467] Processing CreateTablet for tablet dbbabab708864499bddd590b422ce239 (DEFAULT_TABLE table=test-table [id=5cabad6c2d0c4c9b8610e3c70384284b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:32.544668 16566 tablet_service.cc:1467] Processing CreateTablet for tablet dbbabab708864499bddd590b422ce239 (DEFAULT_TABLE table=test-table [id=5cabad6c2d0c4c9b8610e3c70384284b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:32.544899 16566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbabab708864499bddd590b422ce239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.544901 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbabab708864499bddd590b422ce239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.546820 16796 tablet_bootstrap.cc:492] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Bootstrap starting.
I20260620 06:11:32.546833 16801 tablet_bootstrap.cc:492] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Bootstrap starting.
I20260620 06:11:32.546988 16274 catalog_manager.cc:4841] TS 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873): tablet d003f6f1f8c649f8ba3a3d02fdf9795f (table test-table [id=397ac32fda294337a92e58fd9c005501]) successfully deleted
I20260620 06:11:32.544247 16426 tablet_service.cc:1467] Processing CreateTablet for tablet dbbabab708864499bddd590b422ce239 (DEFAULT_TABLE table=test-table [id=5cabad6c2d0c4c9b8610e3c70384284b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:32.547472 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbabab708864499bddd590b422ce239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.547773 16796 tablet_bootstrap.cc:654] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.548069 16801 tablet_bootstrap.cc:654] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.548756 16819 tablet_replica.cc:330] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: stopping tablet replica
I20260620 06:11:32.548909 16796 tablet_bootstrap.cc:492] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: No bootstrap required, opened a new log
I20260620 06:11:32.548981 16796 ts_tablet_manager.cc:1397] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:32.549425 16796 raft_consensus.cc:348] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.549530 16796 raft_consensus.cc:374] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.549563 16796 raft_consensus.cc:729] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6009014754692ab10d1bd6ae353bb, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.549566 16795 tablet_bootstrap.cc:492] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Bootstrap starting.
I20260620 06:11:32.549671 16796 consensus_queue.cc:260] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.549746 16819 raft_consensus.cc:2229] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:32.549911 16796 ts_tablet_manager.cc:1428] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:32.550021 16819 raft_consensus.cc:2258] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:32.550457 16795 tablet_bootstrap.cc:654] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.551455 16819 ts_tablet_manager.cc:1905] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:32.552251 16795 tablet_bootstrap.cc:492] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: No bootstrap required, opened a new log
I20260620 06:11:32.552332 16795 ts_tablet_manager.cc:1397] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:11:32.552726 16795 raft_consensus.cc:348] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.552829 16795 raft_consensus.cc:374] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.552858 16795 raft_consensus.cc:729] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f97e13a1a645e39b80a1083267b24d, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.552963 16795 consensus_queue.cc:260] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.553200 16795 ts_tablet_manager.cc:1428] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:32.543802 16821 log.cc:1196] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/d003f6f1f8c649f8ba3a3d02fdf9795f
I20260620 06:11:32.555079 16821 ts_tablet_manager.cc:1939] T d003f6f1f8c649f8ba3a3d02fdf9795f P a4f97e13a1a645e39b80a1083267b24d: Deleting consensus metadata
I20260620 06:11:32.556083 16274 catalog_manager.cc:4841] TS a4f97e13a1a645e39b80a1083267b24d (127.14.209.129:46865): tablet d003f6f1f8c649f8ba3a3d02fdf9795f (table test-table [id=397ac32fda294337a92e58fd9c005501]) successfully deleted
I20260620 06:11:32.559160 16819 ts_tablet_manager.cc:1918] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:32.559266 16819 log.cc:1196] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/d003f6f1f8c649f8ba3a3d02fdf9795f
I20260620 06:11:32.559588 16819 ts_tablet_manager.cc:1939] T d003f6f1f8c649f8ba3a3d02fdf9795f P 2ab6c020c3c84dc791e5655b2b094b80: Deleting consensus metadata
I20260620 06:11:32.560701 16275 catalog_manager.cc:4841] TS 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347): tablet d003f6f1f8c649f8ba3a3d02fdf9795f (table test-table [id=397ac32fda294337a92e58fd9c005501]) successfully deleted
I20260620 06:11:32.560995 16801 tablet_bootstrap.cc:492] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: No bootstrap required, opened a new log
I20260620 06:11:32.561088 16801 ts_tablet_manager.cc:1397] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260620 06:11:32.561465 16801 raft_consensus.cc:348] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.561553 16801 raft_consensus.cc:374] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.561574 16801 raft_consensus.cc:729] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab6c020c3c84dc791e5655b2b094b80, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.561738 16801 consensus_queue.cc:260] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.562021 16801 ts_tablet_manager.cc:1428] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:32.565708 16803 raft_consensus.cc:482] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:32.565821 16803 raft_consensus.cc:504] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.566865 16803 leader_election.cc:288] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4f97e13a1a645e39b80a1083267b24d (127.14.209.129:46865), 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873)
I20260620 06:11:32.571913 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.567286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46873 (local address 127.14.209.131:41817)
0620 06:11:32.567528 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.567541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.567615 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.568188 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.568191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.568206 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.568298 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.568303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.569294 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.569297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.570532 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.570543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.571534 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.571537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.571637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.571657 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.571720 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.571783 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":22,"thread_start_us":78,"threads_started":1}
I20260620 06:11:32.572537 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.567804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:41817 (local address 127.14.209.130:46873)
0620 06:11:32.567903 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.567907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.567919 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:32.567960 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.567963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.568000 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.568063 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.568429 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.569170 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.570658 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.571317 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.571430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.571479 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.572347 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.572430 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 06:11:32.572635 16493 tablet.cc:2369] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:32.572957 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbabab708864499bddd590b422ce239" candidate_uuid: "2ab6c020c3c84dc791e5655b2b094b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6009014754692ab10d1bd6ae353bb" is_pre_election: true
I20260620 06:11:32.573109 16586 raft_consensus.cc:2454] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ab6c020c3c84dc791e5655b2b094b80 in term 0.
I20260620 06:11:32.573444 16658 leader_election.cc:302] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab6c020c3c84dc791e5655b2b094b80, 90d6009014754692ab10d1bd6ae353bb; no voters:
I20260620 06:11:32.573547 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.567217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:47823 (local address 127.14.209.129:46865)
0620 06:11:32.567326 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.567330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.567573 (+ 243us) server_negotiation.cc:407] Connection header received
0620 06:11:32.567740 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.567744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.567784 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.567828 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:32.568196 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.568966 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.571721 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.572388 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.572473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.572517 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.573361 (+ 844us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.573442 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:32.573685 16803 raft_consensus.cc:2781] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:32.573742 16803 raft_consensus.cc:482] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:32.573792 16803 raft_consensus.cc:3037] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.573922 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.567098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46865 (local address 127.14.209.131:47823)
0620 06:11:32.567499 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.567513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.567633 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.567955 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.567958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.567973 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:32.568086 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.568092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.569102 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.569105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.571586 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.571595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.573125 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.573128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.573231 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.573252 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.573697 (+ 445us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.573765 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":6}
I20260620 06:11:32.574937 16803 raft_consensus.cc:504] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.575445 16803 leader_election.cc:288] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [CANDIDATE]: Term 1 election: Requested vote from peers a4f97e13a1a645e39b80a1083267b24d (127.14.209.129:46865), 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873)
I20260620 06:11:32.575541 16446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbabab708864499bddd590b422ce239" candidate_uuid: "2ab6c020c3c84dc791e5655b2b094b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f97e13a1a645e39b80a1083267b24d" is_pre_election: true
I20260620 06:11:32.575734 16446 raft_consensus.cc:2454] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ab6c020c3c84dc791e5655b2b094b80 in term 0.
I20260620 06:11:32.575947 16445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbabab708864499bddd590b422ce239" candidate_uuid: "2ab6c020c3c84dc791e5655b2b094b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f97e13a1a645e39b80a1083267b24d"
I20260620 06:11:32.576076 16445 raft_consensus.cc:3037] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.576911 16445 raft_consensus.cc:2454] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ab6c020c3c84dc791e5655b2b094b80 in term 1.
I20260620 06:11:32.577139 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbabab708864499bddd590b422ce239" candidate_uuid: "2ab6c020c3c84dc791e5655b2b094b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6009014754692ab10d1bd6ae353bb"
I20260620 06:11:32.577176 16658 leader_election.cc:302] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab6c020c3c84dc791e5655b2b094b80, a4f97e13a1a645e39b80a1083267b24d; no voters:
I20260620 06:11:32.577270 16586 raft_consensus.cc:3037] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.578464 16586 raft_consensus.cc:2454] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ab6c020c3c84dc791e5655b2b094b80 in term 1.
I20260620 06:11:32.578850 16803 raft_consensus.cc:2781] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:32.579030 16803 raft_consensus.cc:686] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 LEADER]: Becoming Leader. State: Replica: 2ab6c020c3c84dc791e5655b2b094b80, State: Running, Role: LEADER
I20260620 06:11:32.579304 16803 consensus_queue.cc:237] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.581629 16287 catalog_manager.cc:5495] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "2ab6c020c3c84dc791e5655b2b094b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } health_report { overall_health: HEALTHY } } }
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:32.621354 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.612667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42785 (local address 127.0.0.1:60352)
0620 06:11:32.612935 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.612951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.613057 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.613304 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.613307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.613797 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:32.613907 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.613914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.614614 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.614617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.615549 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.615559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.615974 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.615978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.616063 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.616637 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:32.616651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:32.618176 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:32.619860 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.619871 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.619876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.620093 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.620364 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:32.620367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:32.620369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:32.620450 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:32.620802 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:32.620809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:32.620913 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.621024 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.621167 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
I20260620 06:11:32.621354 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.612731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.14.209.190:42785)
0620 06:11:32.612860 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.612864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.613029 (+ 165us) server_negotiation.cc:407] Connection header received
0620 06:11:32.613147 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.613149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.613187 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.613254 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.614019 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.614512 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.615671 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.615869 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.615955 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.618313 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.618317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.618318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.618337 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.619738 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.620189 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.620195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.620198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.620270 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.620541 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.620545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.620548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.620699 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.620804 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.621077 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.621219 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":55}
I20260620 06:11:32.625125 16287 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:60352:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:11:32.625833 16287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:32.630342 16426 tablet_service.cc:1467] Processing CreateTablet for tablet 8df1b597e42c42a19d49c47ecbd94369 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ddb0ea5a3254ceea86cf418b68afaa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:11:32.630420 16706 tablet_service.cc:1467] Processing CreateTablet for tablet 8df1b597e42c42a19d49c47ecbd94369 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ddb0ea5a3254ceea86cf418b68afaa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:11:32.630522 16566 tablet_service.cc:1467] Processing CreateTablet for tablet 8df1b597e42c42a19d49c47ecbd94369 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ddb0ea5a3254ceea86cf418b68afaa8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:11:32.630707 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df1b597e42c42a19d49c47ecbd94369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.630707 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df1b597e42c42a19d49c47ecbd94369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.630885 16566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df1b597e42c42a19d49c47ecbd94369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:32.633359 16795 tablet_bootstrap.cc:492] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: Bootstrap starting.
I20260620 06:11:32.633400 16801 tablet_bootstrap.cc:492] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: Bootstrap starting.
I20260620 06:11:32.634353 16795 tablet_bootstrap.cc:654] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.634631 16801 tablet_bootstrap.cc:654] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.634917 16796 tablet_bootstrap.cc:492] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: Bootstrap starting.
I20260620 06:11:32.635839 16796 tablet_bootstrap.cc:654] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:32.636461 16801 tablet_bootstrap.cc:492] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: No bootstrap required, opened a new log
I20260620 06:11:32.636547 16801 ts_tablet_manager.cc:1397] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:32.636982 16795 tablet_bootstrap.cc:492] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: No bootstrap required, opened a new log
I20260620 06:11:32.637070 16795 ts_tablet_manager.cc:1397] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:11:32.636996 16801 raft_consensus.cc:348] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.637115 16801 raft_consensus.cc:374] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.637156 16801 raft_consensus.cc:729] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab6c020c3c84dc791e5655b2b094b80, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.637269 16801 consensus_queue.cc:260] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.637460 16795 raft_consensus.cc:348] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.637570 16795 raft_consensus.cc:374] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.637593 16801 ts_tablet_manager.cc:1428] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:32.637612 16795 raft_consensus.cc:729] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f97e13a1a645e39b80a1083267b24d, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.637737 16795 consensus_queue.cc:260] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.637768 16803 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.637869 16803 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.638026 16795 ts_tablet_manager.cc:1428] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:32.638562 16796 tablet_bootstrap.cc:492] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: No bootstrap required, opened a new log
I20260620 06:11:32.638648 16796 ts_tablet_manager.cc:1397] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:11:32.638976 16796 raft_consensus.cc:348] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.639119 16796 raft_consensus.cc:374] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:32.639165 16796 raft_consensus.cc:729] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6009014754692ab10d1bd6ae353bb, State: Initialized, Role: FOLLOWER
I20260620 06:11:32.639297 16796 consensus_queue.cc:260] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.639590 16796 ts_tablet_manager.cc:1428] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:32.640024 16799 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.640154 16799 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.640610 16816 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.640736 16816 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.686048 16816 raft_consensus.cc:482] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:32.686218 16816 raft_consensus.cc:504] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.686700 16816 leader_election.cc:288] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873), 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347)
I20260620 06:11:32.687115 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df1b597e42c42a19d49c47ecbd94369" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6009014754692ab10d1bd6ae353bb" is_pre_election: true
I20260620 06:11:32.687124 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df1b597e42c42a19d49c47ecbd94369" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab6c020c3c84dc791e5655b2b094b80" is_pre_election: true
I20260620 06:11:32.687278 16726 raft_consensus.cc:2454] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 0.
I20260620 06:11:32.687278 16586 raft_consensus.cc:2454] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 0.
I20260620 06:11:32.687561 16379 leader_election.cc:302] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab6c020c3c84dc791e5655b2b094b80, a4f97e13a1a645e39b80a1083267b24d; no voters:
I20260620 06:11:32.687742 16816 raft_consensus.cc:2781] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:32.687846 16816 raft_consensus.cc:482] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:32.687942 16816 raft_consensus.cc:3037] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.688913 16816 raft_consensus.cc:504] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.689251 16816 leader_election.cc:288] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 election: Requested vote from peers 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873), 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347)
I20260620 06:11:32.689587 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df1b597e42c42a19d49c47ecbd94369" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6009014754692ab10d1bd6ae353bb"
I20260620 06:11:32.689561 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df1b597e42c42a19d49c47ecbd94369" candidate_uuid: "a4f97e13a1a645e39b80a1083267b24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab6c020c3c84dc791e5655b2b094b80"
I20260620 06:11:32.689694 16726 raft_consensus.cc:3037] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.689694 16586 raft_consensus.cc:3037] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:32.690739 16586 raft_consensus.cc:2454] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 1.
I20260620 06:11:32.690739 16726 raft_consensus.cc:2454] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f97e13a1a645e39b80a1083267b24d in term 1.
I20260620 06:11:32.691011 16378 leader_election.cc:302] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d6009014754692ab10d1bd6ae353bb, a4f97e13a1a645e39b80a1083267b24d; no voters:
I20260620 06:11:32.691181 16816 raft_consensus.cc:2781] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:32.691296 16816 raft_consensus.cc:686] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [term 1 LEADER]: Becoming Leader. State: Replica: a4f97e13a1a645e39b80a1083267b24d, State: Running, Role: LEADER
I20260620 06:11:32.691432 16816 consensus_queue.cc:237] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } }
I20260620 06:11:32.692157 16806 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: TxnStatusTablet state changed. Reason: New leader a4f97e13a1a645e39b80a1083267b24d. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.692286 16806 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:11:32.692700 16838 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:11:32.692876 16838 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:11:32.693159 16289 catalog_manager.cc:5495] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d reported cstate change: term changed from 0 to 1, leader changed from <none> to a4f97e13a1a645e39b80a1083267b24d (127.14.209.129). New cstate: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } health_report { overall_health: UNKNOWN } } }
I20260620 06:11:32.706806 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.702983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46865 (local address 127.0.0.1:47018)
0620 06:11:32.703217 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.703237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.703331 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.703607 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.703609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.703615 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:32.703699 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.703703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.704709 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.704714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.705554 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.705566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.706116 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.706119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.706190 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.706232 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.706549 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.706561 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.706623 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.706704 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 06:11:32.706809 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.703141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.14.209.129:46865)
0620 06:11:32.703267 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.703271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.703287 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:32.703448 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.703451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.703500 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.703579 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:32.703800 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.704488 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.705702 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.705983 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.706091 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.706365 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.706442 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.706532 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.706637 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.706687 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:11:32.709215 16726 raft_consensus.cc:1261] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Refusing update from remote peer a4f97e13a1a645e39b80a1083267b24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:32.709251 16586 raft_consensus.cc:1261] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Refusing update from remote peer a4f97e13a1a645e39b80a1083267b24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:32.709602 16806 consensus_queue.cc:1035] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:32.709748 16816 consensus_queue.cc:1035] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:32.710996 16799 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: TxnStatusTablet state changed. Reason: New leader a4f97e13a1a645e39b80a1083267b24d. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.711014 16803 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: TxnStatusTablet state changed. Reason: New leader a4f97e13a1a645e39b80a1083267b24d. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.711127 16799 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.711149 16803 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.712628 16816 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.712729 16816 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:11:32.712989 16816 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.713068 16816 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P a4f97e13a1a645e39b80a1083267b24d: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:11:32.713544 16799 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.713567 16809 mvcc.cc:205] Tried to move back new op lower bound from 7298809416532967424 to 7298809416465362944. Current Snapshot: MvccSnapshot[applied={T|T < 7298809416532967424 or (T in {7298809416532967424})}]
I20260620 06:11:32.713645 16799 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P 90d6009014754692ab10d1bd6ae353bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.716490 16803 tablet_replica.cc:439] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4f97e13a1a645e39b80a1083267b24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 } } peers { permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 } } peers { permanent_uuid: "2ab6c020c3c84dc791e5655b2b094b80" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 37347 } } }
I20260620 06:11:32.716583 16803 tablet_replica.cc:442] T 8df1b597e42c42a19d49c47ecbd94369 P 2ab6c020c3c84dc791e5655b2b094b80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:32.722936 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.718520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.14.209.131:37347)
0620 06:11:32.718639 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.718642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.718654 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:32.718794 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.718796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.718840 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.718946 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:32.719224 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.719797 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.721247 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.721526 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.721613 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.721899 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.722525 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.722610 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.722712 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.722756 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:32.732532 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.729257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46865 (local address 127.14.209.131:48175)
0620 06:11:32.729379 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.729400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.729470 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.729675 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.729676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.729680 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:32.729767 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.729771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.730711 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.730713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.731345 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.731353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.731892 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.731895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.731970 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.732009 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.732305 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.732311 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.732378 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.732439 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 06:11:32.732758 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.729340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:48175 (local address 127.14.209.129:46865)
0620 06:11:32.729454 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.729459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.729472 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:32.729546 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.729549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.729592 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.729673 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:32.729865 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.730605 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.731453 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.731782 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.731875 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.732124 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.732195 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.732268 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.732411 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.732464 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:32.738619 16586 raft_consensus.cc:1261] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Refusing update from remote peer 2ab6c020c3c84dc791e5655b2b094b80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:32.738862 16445 raft_consensus.cc:1261] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Refusing update from remote peer 2ab6c020c3c84dc791e5655b2b094b80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:32.739086 16826 consensus_queue.cc:1035] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6009014754692ab10d1bd6ae353bb" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:32.739363 16803 consensus_queue.cc:1035] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4f97e13a1a645e39b80a1083267b24d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:32.761672 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.757140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:37419 (local address 127.14.209.131:37347)
0620 06:11:32.757243 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.757247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.757450 (+ 203us) server_negotiation.cc:407] Connection header received
0620 06:11:32.757534 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.757537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.757581 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.757660 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:32.757946 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.758707 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.760590 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.760930 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.761036 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.761276 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.761353 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.761431 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.761498 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.761546 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:11:32.761687 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.757083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:37347 (local address 127.14.209.129:37419)
0620 06:11:32.757360 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:32.757381 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:32.757466 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:32.757718 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:32.757720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:32.757728 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:32.757839 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.757844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.759209 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.759215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.760421 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:32.760438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.761041 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.761044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:32.761114 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.761158 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:32.761434 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:32.761439 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:11:32.761488 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 06:11:32.761564 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260620 06:11:32.783505 16289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:32.783663 16289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:32.786022 16289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d43da5d67bfc4762af37363585949a87: Sending DeleteTablet for 3 replicas of tablet dbbabab708864499bddd590b422ce239
I20260620 06:11:32.786691 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbabab708864499bddd590b422ce239 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:32 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:46992
I20260620 06:11:32.786857 16566 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbabab708864499bddd590b422ce239 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:32 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:39006
I20260620 06:11:32.787014 16821 tablet_replica.cc:330] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: stopping tablet replica
I20260620 06:11:32.787127 16820 tablet_replica.cc:330] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: stopping tablet replica
I20260620 06:11:32.787238 16820 raft_consensus.cc:2229] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:32.787341 16820 raft_consensus.cc:2258] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:32.787518 16821 raft_consensus.cc:2229] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:32.787635 16821 raft_consensus.cc:2258] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:32.787830 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbabab708864499bddd590b422ce239 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:32 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:33788
I20260620 06:11:32.787973 16819 tablet_replica.cc:330] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: stopping tablet replica
I20260620 06:11:32.788058 16819 raft_consensus.cc:2229] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:32.788147 16820 ts_tablet_manager.cc:1905] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:32.788298 16819 raft_consensus.cc:2258] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:32.789016 16821 ts_tablet_manager.cc:1905] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:32.791580 16821 ts_tablet_manager.cc:1918] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:32.791659 16821 log.cc:1196] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/dbbabab708864499bddd590b422ce239
I20260620 06:11:32.791920 16821 ts_tablet_manager.cc:1939] T dbbabab708864499bddd590b422ce239 P a4f97e13a1a645e39b80a1083267b24d: Deleting consensus metadata
I20260620 06:11:32.791939 16820 ts_tablet_manager.cc:1918] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:32.792138 16820 log.cc:1196] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/dbbabab708864499bddd590b422ce239
I20260620 06:11:32.792522 16820 ts_tablet_manager.cc:1939] T dbbabab708864499bddd590b422ce239 P 90d6009014754692ab10d1bd6ae353bb: Deleting consensus metadata
I20260620 06:11:32.792536 16274 catalog_manager.cc:4841] TS a4f97e13a1a645e39b80a1083267b24d (127.14.209.129:46865): tablet dbbabab708864499bddd590b422ce239 (table test-table [id=5cabad6c2d0c4c9b8610e3c70384284b]) successfully deleted
I20260620 06:11:32.793200 16819 ts_tablet_manager.cc:1905] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:32.795776 16819 ts_tablet_manager.cc:1918] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:32.795854 16819 log.cc:1196] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/dbbabab708864499bddd590b422ce239
I20260620 06:11:32.796087 16819 ts_tablet_manager.cc:1939] T dbbabab708864499bddd590b422ce239 P 2ab6c020c3c84dc791e5655b2b094b80: Deleting consensus metadata
I20260620 06:11:32.796634 16274 catalog_manager.cc:4841] TS 90d6009014754692ab10d1bd6ae353bb (127.14.209.130:46873): tablet dbbabab708864499bddd590b422ce239 (table test-table [id=5cabad6c2d0c4c9b8610e3c70384284b]) successfully deleted
I20260620 06:11:32.796716 16275 catalog_manager.cc:4841] TS 2ab6c020c3c84dc791e5655b2b094b80 (127.14.209.131:37347): tablet dbbabab708864499bddd590b422ce239 (table test-table [id=5cabad6c2d0c4c9b8610e3c70384284b]) successfully deleted
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935892, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:32.809908 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.798173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.14.209.129:46865)
0620 06:11:32.798279 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.798283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.798523 (+ 240us) server_negotiation.cc:407] Connection header received
0620 06:11:32.798638 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.798641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.798689 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.798768 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.799717 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.800496 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.801772 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.802109 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.802175 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.806028 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.806038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.806052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.806077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.808067 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.808555 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.808561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.808563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.808652 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.808936 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.808939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.808941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.809223 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.809429 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.809637 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.809750 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 06:11:32.810437 16483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47026
I20260620 06:11:32.821004 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:32.812073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.14.209.190:42785)
0620 06:11:32.812191 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:32.812196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:32.812263 (+ 67us) server_negotiation.cc:407] Connection header received
0620 06:11:32.812368 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:32.812371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:32.812415 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:32.812497 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:32.813195 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.814390 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.815529 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:32.815870 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:32.815975 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:32.817412 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:32.817416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:32.817419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:32.817443 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:32.819189 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.819642 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.819645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.819647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.819732 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:32.820051 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:32.820055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:32.820057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:32.820259 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:32.820361 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:32.820741 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 06:11:32.820851 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:11:32.821702 16289 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60358
I20260620 06:11:32.824219 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16361
I20260620 06:11:32.831809 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16501
I20260620 06:11:32.837778 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16641
I20260620 06:11:32.843699 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16257
2026-06-20T06:11:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2629 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16857](info): set up 2 sockets
Jun 20 06:11:32 dist-test-slave-zt01 krb5kdc[16857](info): commencing operation
krb5kdc: starting...
W20260620 06:11:33.878753 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:33 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:33.994132 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46795
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:34.097231 16870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:34.097466 16870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:34.097504 16870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:34.102744 16870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:34.102805 16870 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:11:34.102857 16870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:34.102885 16870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:34.102909 16870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:34.102939 16870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:34.110086 16870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46795
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:34.111027 16870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:34.112025 16870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:34.120018 16875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.120136 16876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.119994 16878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:34.120714 16870 server_base.cc:1034] running on GCE node
I20260620 06:11:34.121236 16870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:34.122315 16870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:34.123523 16870 hybrid_clock.cc:648] HybridClock initialized: now 1781935894123482 us; error 59 us; skew 500 ppm
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:34.126398 16870 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:34.127447 16870 webserver.cc:460] Webserver started at http://127.14.209.190:42559/ using document root <none> and password file <none>
I20260620 06:11:34.127995 16870 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:34.128093 16870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:34.128315 16870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:34.130298 16870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "355f97f62b234102bd02525c40b87308"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "3afb05044b48c347b25718609d9279a9"
server_key_iv: "d5a4e6989546bb5bd14b52cb078e0488"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.130896 16870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "355f97f62b234102bd02525c40b87308"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "3afb05044b48c347b25718609d9279a9"
server_key_iv: "d5a4e6989546bb5bd14b52cb078e0488"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.134461 16870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:34.136885 16885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.138073 16870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:34.138221 16870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "355f97f62b234102bd02525c40b87308"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "3afb05044b48c347b25718609d9279a9"
server_key_iv: "d5a4e6989546bb5bd14b52cb078e0488"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.138329 16870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:34.183907 16870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:34.186892 16870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:34.187158 16870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:34.194628 16870 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:46795
I20260620 06:11:34.194648 16947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:46795 every 8 connection(s)
I20260620 06:11:34.195622 16870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:34.198314 16948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:34.199654 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16870
I20260620 06:11:34.199772 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:34.200064 15174 external_mini_cluster.cc:1428] Setting key 10d12f2e6162e96d987d324ab7b85383
I20260620 06:11:34.204252 16948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Bootstrap starting.
I20260620 06:11:34.206319 16948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:34.207002 16948 log.cc:824] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:34.208876 16948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: No bootstrap required, opened a new log
I20260620 06:11:34.211658 16948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355f97f62b234102bd02525c40b87308" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46795 } }
I20260620 06:11:34.211899 16948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:34.211978 16948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355f97f62b234102bd02525c40b87308, State: Initialized, Role: FOLLOWER
I20260620 06:11:34.212432 16948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355f97f62b234102bd02525c40b87308" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46795 } }
I20260620 06:11:34.212554 16948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:34.212634 16948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:34.212749 16948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:34.213732 16948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355f97f62b234102bd02525c40b87308" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46795 } }
I20260620 06:11:34.214071 16948 leader_election.cc:302] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 355f97f62b234102bd02525c40b87308; no voters:
I20260620 06:11:34.214366 16948 leader_election.cc:288] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:34.214373 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.201466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.14.209.190:46795)
0620 06:11:34.201884 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.201902 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.201937 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:11:34.202563 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.202579 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.202876 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.203218 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.203971 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.204968 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.205782 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.206239 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.206416 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.209052 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.209057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.209069 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.209094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.211355 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.211790 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.211801 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.211805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.211935 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.212264 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.212267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.212269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.212734 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.212964 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.213187 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.213472 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260620 06:11:34.214453 16953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:34.214630 16953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [term 1 LEADER]: Becoming Leader. State: Replica: 355f97f62b234102bd02525c40b87308, State: Running, Role: LEADER
I20260620 06:11:34.214927 16953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355f97f62b234102bd02525c40b87308" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46795 } }
I20260620 06:11:34.215631 16948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:34.216162 16954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "355f97f62b234102bd02525c40b87308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355f97f62b234102bd02525c40b87308" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46795 } } }
I20260620 06:11:34.216276 16954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:34.216722 16955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355f97f62b234102bd02525c40b87308. Latest consensus state: current_term: 1 leader_uuid: "355f97f62b234102bd02525c40b87308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355f97f62b234102bd02525c40b87308" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46795 } } }
I20260620 06:11:34.216830 16955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308 [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:34.220942 16968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:34.221005 16968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:34.221091 16957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:34.221758 16957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:34.226557 16957 catalog_manager.cc:1347] Generated new cluster ID: 839ab068acdb4d80954274d0d6d18320
I20260620 06:11:34.226621 16957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:34.257928 16957 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:34.258890 16957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:34.263233 16957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Generated new TSK 0
I20260620 06:11:34.263955 16957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:34.301579 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46795
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:34.406749 16974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:34.406993 16974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:34.407038 16974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:34.411216 16974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:34.411273 16974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:34.411311 16974 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:11:34.411331 16974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:11:34.411362 16974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:34.416625 16974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:34.417574 16974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:34.418511 16974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:34.427340 16979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.427385 16982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.427335 16980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:34.427522 16974 server_base.cc:1034] running on GCE node
I20260620 06:11:34.427919 16974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:34.428484 16974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:34.429672 16974 hybrid_clock.cc:648] HybridClock initialized: now 1781935894429655 us; error 32 us; skew 500 ppm
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:34.432325 16974 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:34.433209 16974 webserver.cc:460] Webserver started at http://127.14.209.129:37557/ using document root <none> and password file <none>
I20260620 06:11:34.433729 16974 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:34.433813 16974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:34.434031 16974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:34.435969 16974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "42c34bc9fc3b430b920663c5b46fe34c"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "90ba081084b3aa76437d30a59cf9ca6e"
server_key_iv: "60ddc15e266592f09541d13600abff10"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.436543 16974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "42c34bc9fc3b430b920663c5b46fe34c"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "90ba081084b3aa76437d30a59cf9ca6e"
server_key_iv: "60ddc15e266592f09541d13600abff10"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.439996 16974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:34.442265 16989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.443552 16974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:34.443679 16974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "42c34bc9fc3b430b920663c5b46fe34c"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "90ba081084b3aa76437d30a59cf9ca6e"
server_key_iv: "60ddc15e266592f09541d13600abff10"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.443778 16974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:34.459455 16974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:34.462658 16974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:34.462859 16974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:34.464202 16974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:34.464275 16974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.464337 16974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:34.464377 16974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:34.475381 16974 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:41759
I20260620 06:11:34.475461 17104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:41759 every 8 connection(s)
I20260620 06:11:34.476475 16974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:34.477118 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 16974
I20260620 06:11:34.477229 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:34.477624 15174 external_mini_cluster.cc:1428] Setting key ba90223aae99805c69571a8fb6d3e044
I20260620 06:11:34.477819 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.465097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:40511 (local address 127.14.209.190:46795)
0620 06:11:34.465217 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.465220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.466201 (+ 981us) server_negotiation.cc:407] Connection header received
0620 06:11:34.467362 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.467369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.467440 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.467541 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.468917 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.469606 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.470872 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.471191 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.471289 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.473837 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.473841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.473844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.473876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.475526 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.476050 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.476053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.476054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.476126 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.476490 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.476493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.476494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.476736 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.476841 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.477452 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.477659 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:34.478515 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.465603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.14.209.129:40511)
0620 06:11:34.466055 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.466093 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.467083 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.467852 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.467862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.468187 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:34.468768 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.468791 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.469732 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.469739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.470747 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.470756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.471360 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.471365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.471513 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.472084 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:34.472109 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:34.473681 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:34.475624 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.475630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.475642 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.475949 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.476217 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.476220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.476222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.476387 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.476846 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:34.476853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:34.477167 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.477385 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.477656 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260620 06:11:34.479899 17105 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46795
I20260620 06:11:34.480029 17105 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:34.480573 17105 heartbeater.cc:511] Master 127.14.209.190:46795 requested a full tablet report, sending...
I20260620 06:11:34.482060 16896 ts_manager.cc:194] Registered new tserver with Master: 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759)
I20260620 06:11:34.483652 16896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:40511
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:34.489961 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.481895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.14.209.129:37581)
0620 06:11:34.482135 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.482169 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.482254 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.483471 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.483474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.483661 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:34.483770 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.483778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.484514 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.484517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.485073 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.485080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.485623 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.485629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.485766 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.486316 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:34.486332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:34.486769 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:34.488585 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.488588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.488589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.488763 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.489073 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.489076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.489077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.489191 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.489512 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:34.489513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:34.489579 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.489661 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.489755 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260620 06:11:34.489979 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.483178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:37581 (local address 127.14.209.190:46795)
0620 06:11:34.483291 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.483293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.483304 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:11:34.483334 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.483336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.483367 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.483456 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.483922 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.484395 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.485187 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.485485 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.485587 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.486926 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.486929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.486931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.486950 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.488485 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.488844 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.488847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.488848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.488960 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.489276 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.489278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.489279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.489430 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.489505 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.489713 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.489794 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:34.506948 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46795
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:34.611932 17114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:34.612149 17114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:34.612191 17114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:34.616104 17114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:34.616154 17114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:34.616192 17114 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:11:34.616206 17114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:11:34.616236 17114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:34.621359 17114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:34.622258 17114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:34.623148 17114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:34.632369 17119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.632624 17122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.632665 17120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:34.632754 17114 server_base.cc:1034] running on GCE node
I20260620 06:11:34.633206 17114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:34.633810 17114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:34.634999 17114 hybrid_clock.cc:648] HybridClock initialized: now 1781935894634989 us; error 30 us; skew 500 ppm
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:34.637679 17114 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:34.638746 17114 webserver.cc:460] Webserver started at http://127.14.209.130:46135/ using document root <none> and password file <none>
I20260620 06:11:34.639457 17114 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:34.639549 17114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:34.639776 17114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:34.641846 17114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "572b5470ace247a484b59e0783c54aad"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "9afd01492e36585b6b60044a7eaddd04"
server_key_iv: "9b9ca71f136aba8bfe03213190ceacc5"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.642426 17114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "572b5470ace247a484b59e0783c54aad"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "9afd01492e36585b6b60044a7eaddd04"
server_key_iv: "9b9ca71f136aba8bfe03213190ceacc5"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.646020 17114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:34.648412 17129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.649636 17114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:34.649770 17114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "572b5470ace247a484b59e0783c54aad"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "9afd01492e36585b6b60044a7eaddd04"
server_key_iv: "9b9ca71f136aba8bfe03213190ceacc5"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.649883 17114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:34.674116 17114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:34.676940 17114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:34.677127 17114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:34.678917 17114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:34.678974 17114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.679032 17114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:34.679076 17114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:34.689915 17114 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:33081
I20260620 06:11:34.689968 17244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:33081 every 8 connection(s)
I20260620 06:11:34.690932 17114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:34.692333 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.681066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:38429 (local address 127.14.209.190:46795)
0620 06:11:34.681207 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.681212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.681228 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:34.681918 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.681921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.681976 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.682065 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.683735 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.684612 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.685866 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.686098 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.686212 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.688631 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.688635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.688637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.688665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.690273 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.690782 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.690786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.690788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.690880 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.691279 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.691284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.691288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.691525 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.691630 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.692127 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.692220 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:11:34.692664 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 17114
I20260620 06:11:34.692791 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:34.693080 15174 external_mini_cluster.cc:1428] Setting key b0d72b63041c7271414a2e605487f72e
I20260620 06:11:34.693115 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.680019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.14.209.130:38429)
0620 06:11:34.680811 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.680879 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.681715 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.682274 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.682287 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.682815 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:34.683539 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.683563 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.684743 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.684747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.685738 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.685747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.686256 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.686261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.686429 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.686971 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:34.686989 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:34.688475 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:34.690385 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.690390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.690401 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.690676 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.690977 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.690980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.690981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.691177 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.691686 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:34.691693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:34.691908 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.692047 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.692257 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":122,"threads_started":1}
I20260620 06:11:34.694515 17245 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46795
I20260620 06:11:34.694648 17245 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:34.695194 17245 heartbeater.cc:511] Master 127.14.209.190:46795 requested a full tablet report, sending...
I20260620 06:11:34.696159 16896 ts_manager.cc:194] Registered new tserver with Master: 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081)
I20260620 06:11:34.696931 16896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:38429
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:11:34.704392 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.696756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.14.209.130:45117)
0620 06:11:34.697202 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.697217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.697334 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.697617 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.697622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.697851 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:34.698016 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.698026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.698987 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.698993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.700026 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.700039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.700451 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.700457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.700583 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.701032 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:34.701045 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:34.701452 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:34.703166 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.703172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.703176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.703367 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.703612 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.703614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.703615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.703684 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.704034 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:34.704036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:34.704105 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.704182 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.704273 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":111,"threads_started":1}
I20260620 06:11:34.704396 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.696925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:45117 (local address 127.14.209.190:46795)
0620 06:11:34.697040 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.697044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.697275 (+ 231us) server_negotiation.cc:407] Connection header received
0620 06:11:34.697450 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.697453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.697499 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.697585 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.698147 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.698855 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.700144 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.700338 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.700418 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.701589 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.701592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.701594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.701613 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.703041 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.703452 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.703455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.703456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.703534 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.703768 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.703771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.703772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.703960 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.704031 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.704214 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.704292 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:34.722275 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46795
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:34.827739 17254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:34.827958 17254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:34.828032 17254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:34.832171 17254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:34.832221 17254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:34.832259 17254 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:11:34.832302 17254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:11:34.832376 17254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:34.837493 17254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:34.838384 17254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:34.839375 17254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:34.848862 17262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.848920 17259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:34.848918 17260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:34.849231 17254 server_base.cc:1034] running on GCE node
I20260620 06:11:34.849581 17254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:34.850194 17254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:34.851372 17254 hybrid_clock.cc:648] HybridClock initialized: now 1781935894851358 us; error 25 us; skew 500 ppm
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:34.853947 17254 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:34.854794 17254 webserver.cc:460] Webserver started at http://127.14.209.131:45913/ using document root <none> and password file <none>
I20260620 06:11:34.855391 17254 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:34.855477 17254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:34.855669 17254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:34.857721 17254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "7ea58e5b2416a95cc64f96a61cbdb93e"
server_key_iv: "8a524e3a9e34a6ca538d50a081f0cc68"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.858255 17254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "7ea58e5b2416a95cc64f96a61cbdb93e"
server_key_iv: "8a524e3a9e34a6ca538d50a081f0cc68"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.861763 17254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 06:11:34.864004 17269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.865085 17254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:11:34.865221 17254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
format_stamp: "Formatted at 2026-06-20 06:11:34 on dist-test-slave-zt01"
server_key: "7ea58e5b2416a95cc64f96a61cbdb93e"
server_key_iv: "8a524e3a9e34a6ca538d50a081f0cc68"
server_key_version: "encryptionkey@0"
I20260620 06:11:34.865324 17254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:34.879559 17254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:34.882251 17254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:34.882447 17254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:34.883854 17254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:34.883925 17254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:34.884019 17254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:34.884066 17254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:34.894965 17254 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:45051
I20260620 06:11:34.896039 17254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:34.896791 17384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:45051 every 8 connection(s)
I20260620 06:11:34.897598 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.884599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:42273 (local address 127.14.209.190:46795)
0620 06:11:34.884743 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.884748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.885550 (+ 802us) server_negotiation.cc:407] Connection header received
0620 06:11:34.886475 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.886479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.886542 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.886646 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.887892 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.888766 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.890132 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.890369 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.890485 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.893716 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.893720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.893723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.893749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.895448 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.895902 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.895905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.895907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.895996 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.896362 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.896365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.896367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.896603 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.896709 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.897297 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.897419 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:11:34.897969 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 17254
I20260620 06:11:34.898101 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:34.898401 15174 external_mini_cluster.cc:1428] Setting key 548fa4710e3c8376ec65bc8c36979314
I20260620 06:11:34.898547 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.884972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.14.209.131:42273)
0620 06:11:34.885394 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.885441 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.886227 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.886806 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.886815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.887222 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:34.887749 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.887763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.888918 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.888922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.890007 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.890016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.890525 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.890530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.890709 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.891329 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:34.891352 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:34.893551 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:34.895553 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.895557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.895567 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.895800 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.896117 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.896123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.896125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.896262 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.896762 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:34.896770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:34.897040 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.897230 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.897573 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":124,"threads_started":1}
I20260620 06:11:34.899739 17385 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46795
I20260620 06:11:34.899869 17385 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:34.900354 17385 heartbeater.cc:511] Master 127.14.209.190:46795 requested a full tablet report, sending...
I20260620 06:11:34.901425 16902 ts_manager.cc:194] Registered new tserver with Master: 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:34.902158 16902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:42273
I20260620 06:11:34.902671 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:11:34.910029 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.901898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.14.209.131:48371)
0620 06:11:34.902160 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.902173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.902272 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.902845 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.902848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.903104 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:34.903219 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.903225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.904216 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.904219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.904751 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.904757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.905383 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.905386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.905452 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.905906 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:34.905919 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:34.906213 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:34.908411 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.908418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.908423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.908689 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.908997 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:34.909003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:34.909008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:34.909110 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:34.909621 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:34.909624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:34.909695 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.909788 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.909909 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
I20260620 06:11:34.910181 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.901944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:48371 (local address 127.14.209.190:46795)
0620 06:11:34.902555 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.902559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.902571 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:34.902605 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.902608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.902645 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.902742 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.903348 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.904062 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.904863 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.905251 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.905373 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.906359 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.906366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.906371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.906401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.908283 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.908812 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.908815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.908816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.908893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.909234 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.909240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.909244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.909513 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.909626 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.909873 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.909989 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548}
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:11:34 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:34.928694 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.921268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.14.209.190:46795)
0620 06:11:34.921406 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.921409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.921421 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:34.921450 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.921452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.921482 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.921549 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:34.922224 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.922702 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.923587 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.923782 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.923860 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.926138 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:34.926141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:34.926143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:34.926162 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:34.927358 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.927706 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.927709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.927710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.927773 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:34.928042 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:34.928045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:34.928046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:34.928205 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:34.928320 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.928507 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.928591 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:11:34.931198 16902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:34.933993 16902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:34.947949 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.941699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.0.0.1:55050)
0620 06:11:34.942109 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.942157 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.942269 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.943205 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.943211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.943234 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:34.943376 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.943382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.944909 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.944913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.946223 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.946235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.947397 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.947401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.947519 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.947552 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.947694 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.947809 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":42,"thread_start_us":67,"threads_started":1}
I20260620 06:11:34.947985 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.941808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.14.209.130:33081)
0620 06:11:34.942558 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.942571 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.942596 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:11:34.942676 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.942684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.942923 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.943533 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:34.943684 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.944793 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.946343 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.947276 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.947429 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.947558 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.947725 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.947809 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":107,"threads_started":1}
I20260620 06:11:34.948231 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.943274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.14.209.131:45051)
0620 06:11:34.943578 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.943591 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.943615 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:11:34.943715 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.943718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.943845 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.943994 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:34.944189 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.945089 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.946595 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.947370 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.947517 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.947626 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.947997 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.948096 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":141,"threads_started":1}
I20260620 06:11:34.948254 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.942013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41759 (local address 127.0.0.1:48138)
0620 06:11:34.943263 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.943276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.943368 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.944262 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.944264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.944282 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:34.944357 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.944363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.945548 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.945551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.946770 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.946780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.947816 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.947819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.947922 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.948033 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.948081 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.948135 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":8,"thread_start_us":88,"threads_started":1}
I20260620 06:11:34.948518 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.942749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45051 (local address 127.0.0.1:37406)
0620 06:11:34.943542 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:34.943554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:34.943620 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:34.943985 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:34.943988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:34.944005 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:34.944098 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.944104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.945208 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.945211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.946488 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:34.946495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.947691 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.947694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:34.947801 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.947824 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:11:34.947887 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:11:34.948398 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":512,"thread_start_us":80,"threads_started":1}
I20260620 06:11:34.949210 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:34.942557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.14.209.129:41759)
0620 06:11:34.943854 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:34.943867 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:34.943899 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:11:34.943967 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:34.943972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:34.944132 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:34.944267 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:34.944484 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.945421 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.946928 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:34.947697 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:34.948833 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:34.948944 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:34.948993 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 06:11:34.949081 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":81,"threads_started":1}
I20260620 06:11:34.950862 17179 tablet_service.cc:1467] Processing CreateTablet for tablet 60e25c43616943c7b8e296cef5d5a8d9 (DEFAULT_TABLE table=test-table [id=3b15243fa3e24e7898064010408434b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:34.950959 17319 tablet_service.cc:1467] Processing CreateTablet for tablet 60e25c43616943c7b8e296cef5d5a8d9 (DEFAULT_TABLE table=test-table [id=3b15243fa3e24e7898064010408434b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:34.951458 17030 tablet_service.cc:1467] Processing CreateTablet for tablet 60e25c43616943c7b8e296cef5d5a8d9 (DEFAULT_TABLE table=test-table [id=3b15243fa3e24e7898064010408434b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:34.951897 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e25c43616943c7b8e296cef5d5a8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:34.951964 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e25c43616943c7b8e296cef5d5a8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:34.952292 17030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e25c43616943c7b8e296cef5d5a8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:34.958190 17408 tablet_bootstrap.cc:492] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Bootstrap starting.
I20260620 06:11:34.959452 17409 tablet_bootstrap.cc:492] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Bootstrap starting.
I20260620 06:11:34.960426 17408 tablet_bootstrap.cc:654] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:34.960634 17410 tablet_bootstrap.cc:492] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Bootstrap starting.
I20260620 06:11:34.961357 17409 tablet_bootstrap.cc:654] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:34.961365 17408 log.cc:824] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:34.962055 17409 log.cc:824] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:34.962517 17410 tablet_bootstrap.cc:654] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:34.963335 17410 log.cc:824] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:34.963392 17408 tablet_bootstrap.cc:492] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: No bootstrap required, opened a new log
I20260620 06:11:34.963577 17408 ts_tablet_manager.cc:1397] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:11:34.964246 17409 tablet_bootstrap.cc:492] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: No bootstrap required, opened a new log
I20260620 06:11:34.964413 17409 ts_tablet_manager.cc:1397] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:11:34.967353 17408 raft_consensus.cc:348] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:34.967623 17408 raft_consensus.cc:374] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:34.967679 17408 raft_consensus.cc:729] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Initialized, Role: FOLLOWER
I20260620 06:11:34.968086 17408 consensus_queue.cc:260] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:34.968081 17409 raft_consensus.cc:348] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:34.968294 17409 raft_consensus.cc:374] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:34.968361 17409 raft_consensus.cc:729] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Initialized, Role: FOLLOWER
I20260620 06:11:34.968518 17410 tablet_bootstrap.cc:492] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: No bootstrap required, opened a new log
I20260620 06:11:34.968714 17410 ts_tablet_manager.cc:1397] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 06:11:34.968751 17409 consensus_queue.cc:260] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:34.968845 17245 heartbeater.cc:503] Master 127.14.209.190:46795 was elected leader, sending a full tablet report...
I20260620 06:11:34.969179 17408 ts_tablet_manager.cc:1428] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:11:34.969537 17105 heartbeater.cc:503] Master 127.14.209.190:46795 was elected leader, sending a full tablet report...
I20260620 06:11:34.969599 17409 ts_tablet_manager.cc:1428] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260620 06:11:34.972810 17410 raft_consensus.cc:348] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:34.973050 17410 raft_consensus.cc:374] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:34.973109 17410 raft_consensus.cc:729] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f51b8b43684ecdbf26fb2ea7e538c7, State: Initialized, Role: FOLLOWER
I20260620 06:11:34.973523 17410 consensus_queue.cc:260] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:34.974084 17385 heartbeater.cc:503] Master 127.14.209.190:46795 was elected leader, sending a full tablet report...
I20260620 06:11:34.974380 17410 ts_tablet_manager.cc:1428] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260620 06:11:34.978631 17106 tablet.cc:2369] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:35.079268 17414 raft_consensus.cc:482] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:35.079540 17414 raft_consensus.cc:504] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.080757 17414 leader_election.cc:288] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:35.085093 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.080973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45051 (local address 127.14.209.130:43297)
0620 06:11:35.081156 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.081172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.081267 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.081625 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.081627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.081640 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:35.081774 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.081782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.082651 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.082654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.083843 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.083853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.084669 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.084672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.084755 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.084774 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.084835 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.084927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 06:11:35.085078 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.081165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:43297 (local address 127.14.209.131:45051)
0620 06:11:35.081361 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.081365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.081378 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:35.081432 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.081435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.081508 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.081593 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:35.081902 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.082518 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.083991 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.084552 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.084675 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.084728 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.084881 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.084948 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 06:11:35.085350 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.081122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:37597 (local address 127.14.209.129:41759)
0620 06:11:35.081241 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.081245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.081487 (+ 242us) server_negotiation.cc:407] Connection header received
0620 06:11:35.081564 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.081568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.081618 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.081688 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:35.082319 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.082907 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.084218 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.084793 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.084921 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.084962 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.085152 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.085224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:11:35.085532 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.080991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41759 (local address 127.14.209.130:37597)
0620 06:11:35.081376 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.081394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.081490 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.081715 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.081717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.081729 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:35.082201 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.082207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.083035 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.083039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.084103 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.084110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.084934 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.084937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.085025 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.085045 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.085121 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.085317 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":14,"thread_start_us":134,"threads_started":1}
I20260620 06:11:35.085822 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e25c43616943c7b8e296cef5d5a8d9" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" is_pre_election: true
I20260620 06:11:35.086117 17339 raft_consensus.cc:2454] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 0.
I20260620 06:11:35.086149 17059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e25c43616943c7b8e296cef5d5a8d9" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c34bc9fc3b430b920663c5b46fe34c" is_pre_election: true
I20260620 06:11:35.086410 17059 raft_consensus.cc:2454] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 0.
I20260620 06:11:35.086598 17130 leader_election.cc:302] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572b5470ace247a484b59e0783c54aad, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:35.086858 17414 raft_consensus.cc:2781] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:35.086973 17414 raft_consensus.cc:482] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:35.087033 17414 raft_consensus.cc:3037] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.088238 17414 raft_consensus.cc:504] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.088613 17414 leader_election.cc:288] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 election: Requested vote from peers 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:35.088961 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e25c43616943c7b8e296cef5d5a8d9" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
I20260620 06:11:35.088968 17059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e25c43616943c7b8e296cef5d5a8d9" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c34bc9fc3b430b920663c5b46fe34c"
I20260620 06:11:35.089087 17339 raft_consensus.cc:3037] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.089087 17059 raft_consensus.cc:3037] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.090240 17339 raft_consensus.cc:2454] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 1.
I20260620 06:11:35.090261 17059 raft_consensus.cc:2454] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 1.
I20260620 06:11:35.090520 17130 leader_election.cc:302] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572b5470ace247a484b59e0783c54aad, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:35.090718 17414 raft_consensus.cc:2781] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:35.090961 17414 raft_consensus.cc:686] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 1 LEADER]: Becoming Leader. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Running, Role: LEADER
I20260620 06:11:35.091305 17414 consensus_queue.cc:237] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.094169 16902 catalog_manager.cc:5495] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad reported cstate change: term changed from 0 to 1, leader changed from <none> to 572b5470ace247a484b59e0783c54aad (127.14.209.130). New cstate: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } health_report { overall_health: UNKNOWN } } }
W20260620 06:11:35.147264 17386 tablet.cc:2369] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:11:35 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:35.161268 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.152954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.14.209.190:46795)
0620 06:11:35.153094 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.153097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.153360 (+ 263us) server_negotiation.cc:407] Connection header received
0620 06:11:35.153442 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.153444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.153486 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.153563 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:35.154297 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.154822 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.155670 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.155880 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.155962 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.158206 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:35.158209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:35.158211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:35.158230 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:35.159769 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:35.160217 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:35.160220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:35.160221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:35.160288 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:35.160528 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:35.160530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:35.160531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:35.160679 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:35.160751 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.161019 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.161134 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":44}
I20260620 06:11:35.161271 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.152810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46795 (local address 127.0.0.1:41720)
0620 06:11:35.153216 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.153232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.153362 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.153608 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.153610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.154038 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:35.154186 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.154193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.154943 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.154945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.155562 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.155570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.155985 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.155988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.156049 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.156562 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:35.156574 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:35.158088 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:35.159892 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:35.159900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:35.159905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:35.160129 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:35.160366 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:35.160368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:35.160369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:35.160445 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:35.160763 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:35.160769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:35.160871 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.160966 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.161083 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":124,"threads_started":1}
I20260620 06:11:35.165231 16902 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:41720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:11:35.165853 16902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:35.170588 17319 tablet_service.cc:1467] Processing CreateTablet for tablet e69c7e2ba8b84debbd490228de6667fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e31b2f560ad14f389acdb147a1f44585]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:11:35.170570 17030 tablet_service.cc:1467] Processing CreateTablet for tablet e69c7e2ba8b84debbd490228de6667fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e31b2f560ad14f389acdb147a1f44585]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:11:35.170595 17179 tablet_service.cc:1467] Processing CreateTablet for tablet e69c7e2ba8b84debbd490228de6667fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e31b2f560ad14f389acdb147a1f44585]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:11:35.171288 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69c7e2ba8b84debbd490228de6667fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:35.171289 17030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69c7e2ba8b84debbd490228de6667fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:35.171290 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69c7e2ba8b84debbd490228de6667fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:35.173832 17408 tablet_bootstrap.cc:492] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: Bootstrap starting.
I20260620 06:11:35.173950 17409 tablet_bootstrap.cc:492] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: Bootstrap starting.
I20260620 06:11:35.174155 17410 tablet_bootstrap.cc:492] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: Bootstrap starting.
I20260620 06:11:35.174757 17409 tablet_bootstrap.cc:654] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:35.174854 17408 tablet_bootstrap.cc:654] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:35.175326 17410 tablet_bootstrap.cc:654] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:35.176533 17409 tablet_bootstrap.cc:492] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: No bootstrap required, opened a new log
I20260620 06:11:35.176633 17409 ts_tablet_manager.cc:1397] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:11:35.176671 17410 tablet_bootstrap.cc:492] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: No bootstrap required, opened a new log
I20260620 06:11:35.176756 17410 ts_tablet_manager.cc:1397] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:35.176796 17408 tablet_bootstrap.cc:492] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: No bootstrap required, opened a new log
I20260620 06:11:35.176870 17408 ts_tablet_manager.cc:1397] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:35.177110 17409 raft_consensus.cc:348] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.177222 17409 raft_consensus.cc:374] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:35.177260 17409 raft_consensus.cc:729] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Initialized, Role: FOLLOWER
I20260620 06:11:35.177208 17410 raft_consensus.cc:348] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.177313 17410 raft_consensus.cc:374] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:35.177260 17408 raft_consensus.cc:348] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.177352 17410 raft_consensus.cc:729] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f51b8b43684ecdbf26fb2ea7e538c7, State: Initialized, Role: FOLLOWER
I20260620 06:11:35.177358 17408 raft_consensus.cc:374] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:35.177402 17408 raft_consensus.cc:729] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Initialized, Role: FOLLOWER
I20260620 06:11:35.177392 17409 consensus_queue.cc:260] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.177491 17410 consensus_queue.cc:260] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.177531 17408 consensus_queue.cc:260] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.177654 17409 ts_tablet_manager.cc:1428] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:35.177778 17408 ts_tablet_manager.cc:1428] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:35.177731 17416 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.177853 17416 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:35.177897 17415 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.178016 17415 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:35.177986 17414 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.178071 17410 ts_tablet_manager.cc:1428] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:35.178093 17414 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 06:11:35.192019 17246 tablet.cc:2369] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:35.227274 17414 raft_consensus.cc:482] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:35.227396 17414 raft_consensus.cc:504] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.227787 17414 leader_election.cc:288] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:35.228252 17059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69c7e2ba8b84debbd490228de6667fc" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c34bc9fc3b430b920663c5b46fe34c" is_pre_election: true
I20260620 06:11:35.228260 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69c7e2ba8b84debbd490228de6667fc" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" is_pre_election: true
I20260620 06:11:35.228412 17059 raft_consensus.cc:2454] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 0.
I20260620 06:11:35.228413 17339 raft_consensus.cc:2454] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 0.
I20260620 06:11:35.228705 17133 leader_election.cc:302] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 572b5470ace247a484b59e0783c54aad; no voters:
I20260620 06:11:35.228971 17414 raft_consensus.cc:2781] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:35.229027 17414 raft_consensus.cc:482] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:35.229054 17414 raft_consensus.cc:3037] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.230175 17414 raft_consensus.cc:504] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.230543 17414 leader_election.cc:288] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 election: Requested vote from peers 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:35.230868 17059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69c7e2ba8b84debbd490228de6667fc" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c34bc9fc3b430b920663c5b46fe34c"
I20260620 06:11:35.230904 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69c7e2ba8b84debbd490228de6667fc" candidate_uuid: "572b5470ace247a484b59e0783c54aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
I20260620 06:11:35.231015 17059 raft_consensus.cc:3037] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.231102 17339 raft_consensus.cc:3037] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.231961 17339 raft_consensus.cc:2454] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 1.
I20260620 06:11:35.231980 17059 raft_consensus.cc:2454] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572b5470ace247a484b59e0783c54aad in term 1.
I20260620 06:11:35.232245 17130 leader_election.cc:302] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572b5470ace247a484b59e0783c54aad, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:35.232436 17414 raft_consensus.cc:2781] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:35.232546 17414 raft_consensus.cc:686] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [term 1 LEADER]: Becoming Leader. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Running, Role: LEADER
I20260620 06:11:35.232671 17414 consensus_queue.cc:237] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.233363 17418 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: TxnStatusTablet state changed. Reason: New leader 572b5470ace247a484b59e0783c54aad. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.233481 17418 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:11:35.233757 17430 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:11:35.233992 17430 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:11:35.234382 16898 catalog_manager.cc:5495] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad reported cstate change: term changed from 0 to 1, leader changed from <none> to 572b5470ace247a484b59e0783c54aad (127.14.209.130). New cstate: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } health_report { overall_health: UNKNOWN } } }
I20260620 06:11:35.256278 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.252432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.14.209.130:33081)
0620 06:11:35.252543 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.252553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.252566 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:35.252708 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.252710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.252760 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.252911 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:35.253151 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.253687 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.254902 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.255174 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.255260 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.255571 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.255942 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.256023 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.256134 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.256180 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 06:11:35.256331 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.252289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.0.0.1:55064)
0620 06:11:35.252453 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.252469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.252567 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.252932 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.252935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.252943 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:35.253048 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.253054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.253781 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.253784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.254751 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.254763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.255279 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.255283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.255379 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.255429 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.256030 (+ 601us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.256043 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.256117 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.256198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:11:35.260870 17339 raft_consensus.cc:1261] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Refusing update from remote peer 572b5470ace247a484b59e0783c54aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:35.261008 17059 raft_consensus.cc:1261] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Refusing update from remote peer 572b5470ace247a484b59e0783c54aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:35.261595 17418 consensus_queue.cc:1035] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:35.261747 17414 consensus_queue.cc:1035] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:35.264902 17415 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: TxnStatusTablet state changed. Reason: New leader 572b5470ace247a484b59e0783c54aad. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.265012 17415 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:35.265681 17416 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: TxnStatusTablet state changed. Reason: New leader 572b5470ace247a484b59e0783c54aad. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.265796 17416 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:35.267942 17418 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.268046 17418 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:11:35.268982 17415 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.269085 17415 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 42c34bc9fc3b430b920663c5b46fe34c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:35.269061 17414 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.269150 17414 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 572b5470ace247a484b59e0783c54aad: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:11:35.269728 17435 mvcc.cc:205] Tried to move back new op lower bound from 7298809426982719488 to 7298809426874548224. Current Snapshot: MvccSnapshot[applied={T|T < 7298809426982719488}]
I20260620 06:11:35.271543 17416 tablet_replica.cc:439] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "572b5470ace247a484b59e0783c54aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } } }
I20260620 06:11:35.271646 17416 tablet_replica.cc:442] T e69c7e2ba8b84debbd490228de6667fc P 90f51b8b43684ecdbf26fb2ea7e538c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:11:35.274312 17434 mvcc.cc:205] Tried to move back new op lower bound from 7298809426982719488 to 7298809426874548224. Current Snapshot: MvccSnapshot[applied={T|T < 7298809426982719488 or (T in {7298809426982719488})}]
I20260620 06:11:35.278992 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.275530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.14.209.130:33081)
0620 06:11:35.275875 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.275878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.275888 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:11:35.275919 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.275921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.275954 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.276009 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:35.276280 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.276740 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.277927 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.278133 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.278212 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.278528 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.278612 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.278687 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.278797 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.278836 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260620 06:11:35.289446 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.285945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:45853 (local address 127.14.209.130:33081)
0620 06:11:35.286051 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.286056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.286211 (+ 155us) server_negotiation.cc:407] Connection header received
0620 06:11:35.286308 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.286311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.286355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.286438 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:35.286667 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.287431 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.288279 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.288617 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.288717 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.289023 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.289093 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.289165 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.289242 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.289287 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":46}
I20260620 06:11:35.289454 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.285839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.14.209.130:45853)
0620 06:11:35.286131 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.286144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.286223 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.286452 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.286454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.286460 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:35.286576 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.286580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.287550 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.287553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.288176 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.288184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.288728 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.288730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.288847 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.288886 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.289179 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.289184 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.289233 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.289289 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":19,"thread_start_us":119,"threads_started":1}
I20260620 06:11:35.295810 17059 raft_consensus.cc:1261] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Refusing update from remote peer 572b5470ace247a484b59e0783c54aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:35.295851 17339 raft_consensus.cc:1261] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Refusing update from remote peer 572b5470ace247a484b59e0783c54aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:35.296182 17418 consensus_queue.cc:1035] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:35.296312 17414 consensus_queue.cc:1035] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:35.335690 16898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:35.335863 16898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:35.338074 16898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Sending DeleteTablet for 3 replicas of tablet 60e25c43616943c7b8e296cef5d5a8d9
I20260620 06:11:35.338629 17179 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e25c43616943c7b8e296cef5d5a8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:35 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:55050
I20260620 06:11:35.338737 17030 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e25c43616943c7b8e296cef5d5a8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:35 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:48138
I20260620 06:11:35.339215 17451 tablet_replica.cc:330] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: stopping tablet replica
I20260620 06:11:35.339495 17451 raft_consensus.cc:2229] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:35.339548 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e25c43616943c7b8e296cef5d5a8d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:35 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:37406
I20260620 06:11:35.339597 16898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:11:35.339687 17451 raft_consensus.cc:2258] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 06:11:35.339974 16898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:35.340211 17452 tablet_replica.cc:330] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: stopping tablet replica
I20260620 06:11:35.340533 17452 raft_consensus.cc:2229] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:35.340772 17452 raft_consensus.cc:2258] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:35.340907 17451 ts_tablet_manager.cc:1905] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:35.342160 17452 ts_tablet_manager.cc:1905] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:35.344081 17451 ts_tablet_manager.cc:1918] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:35.344174 17451 log.cc:1196] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/60e25c43616943c7b8e296cef5d5a8d9
I20260620 06:11:35.344482 17451 ts_tablet_manager.cc:1939] T 60e25c43616943c7b8e296cef5d5a8d9 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting consensus metadata
I20260620 06:11:35.344823 17179 tablet_service.cc:1467] Processing CreateTablet for tablet 97ad543c3e1e4c89921462ab5fe805d0 (DEFAULT_TABLE table=test-table [id=da4c9ceeb50b402ca178185bd5d98285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:35.345095 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ad543c3e1e4c89921462ab5fe805d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:35.345538 17452 ts_tablet_manager.cc:1918] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:35.345633 17452 log.cc:1196] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/60e25c43616943c7b8e296cef5d5a8d9
I20260620 06:11:35.345929 17452 ts_tablet_manager.cc:1939] T 60e25c43616943c7b8e296cef5d5a8d9 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting consensus metadata
I20260620 06:11:35.347380 16889 catalog_manager.cc:4841] TS 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759): tablet 60e25c43616943c7b8e296cef5d5a8d9 (table test-table [id=3b15243fa3e24e7898064010408434b1]) successfully deleted
I20260620 06:11:35.347627 17030 tablet_service.cc:1467] Processing CreateTablet for tablet 97ad543c3e1e4c89921462ab5fe805d0 (DEFAULT_TABLE table=test-table [id=da4c9ceeb50b402ca178185bd5d98285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:35.347713 17408 tablet_bootstrap.cc:492] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Bootstrap starting.
I20260620 06:11:35.347856 17030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ad543c3e1e4c89921462ab5fe805d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:35.347873 17319 tablet_service.cc:1467] Processing CreateTablet for tablet 97ad543c3e1e4c89921462ab5fe805d0 (DEFAULT_TABLE table=test-table [id=da4c9ceeb50b402ca178185bd5d98285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:35.348080 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ad543c3e1e4c89921462ab5fe805d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:35.348510 17408 tablet_bootstrap.cc:654] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:35.349737 17408 tablet_bootstrap.cc:492] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: No bootstrap required, opened a new log
I20260620 06:11:35.349812 17408 ts_tablet_manager.cc:1397] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:35.350050 17409 tablet_bootstrap.cc:492] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Bootstrap starting.
I20260620 06:11:35.350210 17410 tablet_bootstrap.cc:492] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Bootstrap starting.
I20260620 06:11:35.350257 16886 catalog_manager.cc:4841] TS 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051): tablet 60e25c43616943c7b8e296cef5d5a8d9 (table test-table [id=3b15243fa3e24e7898064010408434b1]) successfully deleted
I20260620 06:11:35.350245 17408 raft_consensus.cc:348] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.350366 17408 raft_consensus.cc:374] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:35.350402 17408 raft_consensus.cc:729] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Initialized, Role: FOLLOWER
I20260620 06:11:35.350525 17408 consensus_queue.cc:260] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.350808 17408 ts_tablet_manager.cc:1428] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:35.350894 17450 tablet_replica.cc:330] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: stopping tablet replica
I20260620 06:11:35.350944 17409 tablet_bootstrap.cc:654] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:35.351078 17410 tablet_bootstrap.cc:654] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:35.351145 17450 raft_consensus.cc:2229] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:35.351455 17450 raft_consensus.cc:2258] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:35.352638 17450 ts_tablet_manager.cc:1905] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:35.353394 17410 tablet_bootstrap.cc:492] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: No bootstrap required, opened a new log
I20260620 06:11:35.353474 17410 ts_tablet_manager.cc:1397] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 06:11:35.354200 17410 raft_consensus.cc:348] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.354300 17410 raft_consensus.cc:374] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:35.354342 17410 raft_consensus.cc:729] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f51b8b43684ecdbf26fb2ea7e538c7, State: Initialized, Role: FOLLOWER
I20260620 06:11:35.354460 17410 consensus_queue.cc:260] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.354717 17410 ts_tablet_manager.cc:1428] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:11:35.354882 17409 tablet_bootstrap.cc:492] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: No bootstrap required, opened a new log
I20260620 06:11:35.354959 17409 ts_tablet_manager.cc:1397] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 06:11:35.355432 17450 ts_tablet_manager.cc:1918] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:35.355387 17409 raft_consensus.cc:348] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.355496 17409 raft_consensus.cc:374] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:35.355521 17450 log.cc:1196] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/60e25c43616943c7b8e296cef5d5a8d9
I20260620 06:11:35.355530 17409 raft_consensus.cc:729] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Initialized, Role: FOLLOWER
I20260620 06:11:35.355648 17409 consensus_queue.cc:260] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.355855 17450 ts_tablet_manager.cc:1939] T 60e25c43616943c7b8e296cef5d5a8d9 P 572b5470ace247a484b59e0783c54aad: Deleting consensus metadata
I20260620 06:11:35.355895 17409 ts_tablet_manager.cc:1428] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:35.357681 16887 catalog_manager.cc:4841] TS 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081): tablet 60e25c43616943c7b8e296cef5d5a8d9 (table test-table [id=3b15243fa3e24e7898064010408434b1]) successfully deleted
I20260620 06:11:35.496580 17415 raft_consensus.cc:482] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:35.496747 17415 raft_consensus.cc:504] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.497820 17415 leader_election.cc:288] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:35.502900 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.498202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:55911 (local address 127.14.209.130:33081)
0620 06:11:35.498323 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.498328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.498736 (+ 408us) server_negotiation.cc:407] Connection header received
0620 06:11:35.498911 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.498915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.498962 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.499009 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:35.499354 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.499835 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.501651 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.502245 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.502380 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.502435 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.502700 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.502771 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:35.502897 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.498044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.14.209.129:55911)
0620 06:11:35.498628 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.498642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.498759 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.499165 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.499167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.499181 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:35.499247 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.499251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.499950 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.499955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.501513 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.501519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.502385 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.502388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.502454 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.502475 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.502666 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.502736 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":459,"thread_start_us":117,"threads_started":1}
I20260620 06:11:35.503679 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.498154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:37969 (local address 127.14.209.131:45051)
0620 06:11:35.498280 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.498285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.498466 (+ 181us) server_negotiation.cc:407] Connection header received
0620 06:11:35.498675 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.498678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.498737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.498829 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:35.499117 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.499880 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.502506 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.503133 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.503252 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.503299 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.503499 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.503575 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:11:35.503681 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.498041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45051 (local address 127.14.209.129:37969)
0620 06:11:35.498362 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.498374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.498512 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.498850 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.498856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.498875 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:35.498985 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.498994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.500028 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.500031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.501507 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.501519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.503271 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.503275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.503374 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.503391 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.503483 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.503546 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":108,"thread_start_us":82,"threads_started":1}
I20260620 06:11:35.503803 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ad543c3e1e4c89921462ab5fe805d0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572b5470ace247a484b59e0783c54aad" is_pre_election: true
I20260620 06:11:35.504031 17199 raft_consensus.cc:2454] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 0.
I20260620 06:11:35.504339 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ad543c3e1e4c89921462ab5fe805d0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" is_pre_election: true
I20260620 06:11:35.504492 17339 raft_consensus.cc:2454] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 0.
I20260620 06:11:35.504490 16991 leader_election.cc:302] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 572b5470ace247a484b59e0783c54aad; no voters:
I20260620 06:11:35.504791 17415 raft_consensus.cc:2781] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:35.504858 17415 raft_consensus.cc:482] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:35.504899 17415 raft_consensus.cc:3037] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.506146 17415 raft_consensus.cc:504] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.506572 17415 leader_election.cc:288] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 election: Requested vote from peers 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:35.507084 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ad543c3e1e4c89921462ab5fe805d0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
I20260620 06:11:35.507200 17339 raft_consensus.cc:3037] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.508353 17339 raft_consensus.cc:2454] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 1.
I20260620 06:11:35.508700 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ad543c3e1e4c89921462ab5fe805d0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572b5470ace247a484b59e0783c54aad"
I20260620 06:11:35.508823 17199 raft_consensus.cc:3037] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:35.509075 16990 leader_election.cc:302] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:35.509892 17199 raft_consensus.cc:2454] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 1.
I20260620 06:11:35.510183 17415 raft_consensus.cc:2781] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:35.510377 17415 raft_consensus.cc:686] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 LEADER]: Becoming Leader. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Running, Role: LEADER
I20260620 06:11:35.510628 17415 consensus_queue.cc:237] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:35.513276 16896 catalog_manager.cc:5495] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c reported cstate change: term changed from 0 to 1, leader changed from <none> to 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129). New cstate: current_term: 1 leader_uuid: "42c34bc9fc3b430b920663c5b46fe34c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } health_report { overall_health: UNKNOWN } } }
I20260620 06:11:35.560784 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.557347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.14.209.129:41759)
0620 06:11:35.557485 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.557489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.557501 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:35.557588 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.557590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.557637 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.557725 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:35.557991 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.558613 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.559523 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.559733 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.559818 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.560048 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.560496 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.560603 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.560633 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.560677 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:11:35.570205 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.567210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:47329 (local address 127.14.209.130:33081)
0620 06:11:35.567322 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.567325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.567536 (+ 211us) server_negotiation.cc:407] Connection header received
0620 06:11:35.567635 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.567638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.567688 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.567751 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:35.567962 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.568440 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.569272 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.569479 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.569552 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.569823 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.569902 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.569959 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.570065 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.570102 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:35.570227 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.567174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.14.209.129:47329)
0620 06:11:35.567462 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.567475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.567558 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.567776 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.567778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.567783 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:35.567878 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.567882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.568520 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.568522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.569180 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.569189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.569573 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.569575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.569640 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.569677 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.569996 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.570002 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.570051 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.570107 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260620 06:11:35.576018 17339 raft_consensus.cc:1261] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Refusing update from remote peer 42c34bc9fc3b430b920663c5b46fe34c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:35.576333 17199 raft_consensus.cc:1261] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Refusing update from remote peer 42c34bc9fc3b430b920663c5b46fe34c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:35.576475 17415 consensus_queue.cc:1035] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:35.576838 17458 consensus_queue.cc:1035] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:35.593516 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.590369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:50423 (local address 127.14.209.129:41759)
0620 06:11:35.590477 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:35.590481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:35.590531 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:11:35.590677 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:35.590680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:35.590726 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:35.590812 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:35.591067 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.591753 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.592524 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.592728 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.592806 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.593097 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.593178 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.593247 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:35.593351 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 06:11:35.593401 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:11:35.593524 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:35.590286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41759 (local address 127.14.209.130:50423)
0620 06:11:35.590466 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:35.590482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:35.590566 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:35.590813 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:35.590815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:35.590820 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:35.590945 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.590953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.591861 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.591864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.592412 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:35.592420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:35.592821 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:35.592843 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:35.592923 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:35.592971 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:35.593262 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:35.593270 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:11:35.593339 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:11:35.593410 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 06:11:35.594211 17435 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:11:37.150977 16898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:37.151144 16898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:37.153404 16898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Sending DeleteTablet for 3 replicas of tablet 97ad543c3e1e4c89921462ab5fe805d0
I20260620 06:11:37.154048 17030 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ad543c3e1e4c89921462ab5fe805d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:37 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:48138
I20260620 06:11:37.154081 17179 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ad543c3e1e4c89921462ab5fe805d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:37 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:55050
I20260620 06:11:37.154189 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ad543c3e1e4c89921462ab5fe805d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:37 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:37406
I20260620 06:11:37.154506 17492 tablet_replica.cc:330] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: stopping tablet replica
I20260620 06:11:37.154511 17491 tablet_replica.cc:330] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: stopping tablet replica
I20260620 06:11:37.154534 17490 tablet_replica.cc:330] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: stopping tablet replica
I20260620 06:11:37.154639 17492 raft_consensus.cc:2229] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:37.154664 17490 raft_consensus.cc:2229] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:37.154680 17491 raft_consensus.cc:2229] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:37.154743 17492 raft_consensus.cc:2258] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:37.154781 17491 raft_consensus.cc:2258] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:37.155217 17490 raft_consensus.cc:2258] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:37.155488 17492 ts_tablet_manager.cc:1905] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:37.155628 17491 ts_tablet_manager.cc:1905] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:37.158571 17490 ts_tablet_manager.cc:1905] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:37.158607 17492 ts_tablet_manager.cc:1918] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:11:37.158677 17492 log.cc:1196] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/97ad543c3e1e4c89921462ab5fe805d0
I20260620 06:11:37.158699 17491 ts_tablet_manager.cc:1918] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:11:37.158769 17491 log.cc:1196] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/97ad543c3e1e4c89921462ab5fe805d0
I20260620 06:11:37.158958 17492 ts_tablet_manager.cc:1939] T 97ad543c3e1e4c89921462ab5fe805d0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting consensus metadata
I20260620 06:11:37.159132 17491 ts_tablet_manager.cc:1939] T 97ad543c3e1e4c89921462ab5fe805d0 P 572b5470ace247a484b59e0783c54aad: Deleting consensus metadata
I20260620 06:11:37.159662 16886 catalog_manager.cc:4841] TS 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051): tablet 97ad543c3e1e4c89921462ab5fe805d0 (table test-table [id=da4c9ceeb50b402ca178185bd5d98285]) successfully deleted
I20260620 06:11:37.159838 16887 catalog_manager.cc:4841] TS 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081): tablet 97ad543c3e1e4c89921462ab5fe805d0 (table test-table [id=da4c9ceeb50b402ca178185bd5d98285]) successfully deleted
I20260620 06:11:37.161934 17490 ts_tablet_manager.cc:1918] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:11:37.162026 17490 log.cc:1196] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/97ad543c3e1e4c89921462ab5fe805d0
I20260620 06:11:37.162400 17490 ts_tablet_manager.cc:1939] T 97ad543c3e1e4c89921462ab5fe805d0 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting consensus metadata
I20260620 06:11:37.163154 16889 catalog_manager.cc:4841] TS 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759): tablet 97ad543c3e1e4c89921462ab5fe805d0 (table test-table [id=da4c9ceeb50b402ca178185bd5d98285]) successfully deleted
Jun 20 06:11:37 dist-test-slave-zt01 krb5kdc[16857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935897, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:11:37 dist-test-slave-zt01 krb5kdc[16857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935897, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:37.186316 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.173237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.14.209.190:46795)
0620 06:11:37.173498 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.173508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.173608 (+ 100us) server_negotiation.cc:407] Connection header received
0620 06:11:37.173777 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.173780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.173835 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.173934 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:37.174576 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.175288 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.176137 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.176380 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.176489 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.183289 (+ 6800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:37.183293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:37.183295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:37.183319 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:37.184615 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:37.185041 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:37.185046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:37.185048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:37.185129 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:37.185410 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:37.185414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:37.185416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:37.185596 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:37.185736 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.185998 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.186153 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260620 06:11:37.188903 16898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:37.189285 16898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:37.193835 17030 tablet_service.cc:1467] Processing CreateTablet for tablet dc388347d50d4b01af1018bbe8c76cf0 (DEFAULT_TABLE table=test-table [id=63e442c62f3348fb8f1e785f4f6de536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:37.193869 17179 tablet_service.cc:1467] Processing CreateTablet for tablet dc388347d50d4b01af1018bbe8c76cf0 (DEFAULT_TABLE table=test-table [id=63e442c62f3348fb8f1e785f4f6de536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:37.194064 17319 tablet_service.cc:1467] Processing CreateTablet for tablet dc388347d50d4b01af1018bbe8c76cf0 (DEFAULT_TABLE table=test-table [id=63e442c62f3348fb8f1e785f4f6de536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:37.194130 17030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc388347d50d4b01af1018bbe8c76cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:37.194209 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc388347d50d4b01af1018bbe8c76cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:37.194317 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc388347d50d4b01af1018bbe8c76cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:37.196892 17501 tablet_bootstrap.cc:492] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Bootstrap starting.
I20260620 06:11:37.196897 17502 tablet_bootstrap.cc:492] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Bootstrap starting.
I20260620 06:11:37.197712 17501 tablet_bootstrap.cc:654] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:37.197796 17502 tablet_bootstrap.cc:654] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:37.198278 17503 tablet_bootstrap.cc:492] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Bootstrap starting.
I20260620 06:11:37.198972 17502 tablet_bootstrap.cc:492] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: No bootstrap required, opened a new log
I20260620 06:11:37.199199 17502 ts_tablet_manager.cc:1397] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:37.199263 17503 tablet_bootstrap.cc:654] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:37.199648 17502 raft_consensus.cc:348] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.199777 17502 raft_consensus.cc:374] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:37.199822 17502 raft_consensus.cc:729] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Initialized, Role: FOLLOWER
I20260620 06:11:37.199955 17502 consensus_queue.cc:260] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.200436 17503 tablet_bootstrap.cc:492] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: No bootstrap required, opened a new log
I20260620 06:11:37.200519 17503 ts_tablet_manager.cc:1397] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:11:37.200700 17502 ts_tablet_manager.cc:1428] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:37.200948 17503 raft_consensus.cc:348] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.201048 17503 raft_consensus.cc:374] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:37.201089 17503 raft_consensus.cc:729] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f51b8b43684ecdbf26fb2ea7e538c7, State: Initialized, Role: FOLLOWER
I20260620 06:11:37.201216 17503 consensus_queue.cc:260] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.201617 17503 ts_tablet_manager.cc:1428] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:11:37.202670 17501 tablet_bootstrap.cc:492] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: No bootstrap required, opened a new log
I20260620 06:11:37.202798 17501 ts_tablet_manager.cc:1397] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 06:11:37.203233 17501 raft_consensus.cc:348] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.203351 17501 raft_consensus.cc:374] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:37.203392 17501 raft_consensus.cc:729] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Initialized, Role: FOLLOWER
I20260620 06:11:37.203529 17501 consensus_queue.cc:260] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.203830 17501 ts_tablet_manager.cc:1428] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:37.334436 17485 raft_consensus.cc:482] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:37.334587 17485 raft_consensus.cc:504] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.335048 17485 leader_election.cc:288] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:37.335533 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc388347d50d4b01af1018bbe8c76cf0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" is_pre_election: true
I20260620 06:11:37.335683 17339 raft_consensus.cc:2454] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 0.
I20260620 06:11:37.335692 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc388347d50d4b01af1018bbe8c76cf0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572b5470ace247a484b59e0783c54aad" is_pre_election: true
I20260620 06:11:37.335901 17199 raft_consensus.cc:2454] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 0.
I20260620 06:11:37.336052 16990 leader_election.cc:302] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:37.336234 17485 raft_consensus.cc:2781] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:37.336308 17485 raft_consensus.cc:482] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:37.336361 17485 raft_consensus.cc:3037] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:37.337352 17485 raft_consensus.cc:504] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.337685 17485 leader_election.cc:288] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 election: Requested vote from peers 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081), 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051)
I20260620 06:11:37.337967 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc388347d50d4b01af1018bbe8c76cf0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572b5470ace247a484b59e0783c54aad"
I20260620 06:11:37.338085 17199 raft_consensus.cc:3037] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:37.338038 17339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc388347d50d4b01af1018bbe8c76cf0" candidate_uuid: "42c34bc9fc3b430b920663c5b46fe34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7"
I20260620 06:11:37.338162 17339 raft_consensus.cc:3037] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:37.339031 17339 raft_consensus.cc:2454] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 1.
I20260620 06:11:37.339121 17199 raft_consensus.cc:2454] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c34bc9fc3b430b920663c5b46fe34c in term 1.
I20260620 06:11:37.339354 16990 leader_election.cc:302] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:37.339533 17485 raft_consensus.cc:2781] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:37.339645 17485 raft_consensus.cc:686] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 LEADER]: Becoming Leader. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Running, Role: LEADER
I20260620 06:11:37.339787 17485 consensus_queue.cc:237] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.341342 16896 catalog_manager.cc:5495] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c reported cstate change: term changed from 0 to 1, leader changed from <none> to 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129). New cstate: current_term: 1 leader_uuid: "42c34bc9fc3b430b920663c5b46fe34c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } health_report { overall_health: UNKNOWN } } }
I20260620 06:11:37.359591 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.356491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.14.209.129:41759)
0620 06:11:37.356744 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.356751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.356763 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:37.356799 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.356801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.356839 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.356910 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:37.357161 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.357684 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.358582 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.358795 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.358875 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.359184 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:37.359246 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:37.359300 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.359444 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.359485 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":117,"threads_started":1}
I20260620 06:11:37.364835 17199 raft_consensus.cc:1261] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Refusing update from remote peer 42c34bc9fc3b430b920663c5b46fe34c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:37.364933 17339 raft_consensus.cc:1261] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Refusing update from remote peer 42c34bc9fc3b430b920663c5b46fe34c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:37.365195 17481 consensus_queue.cc:1035] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:37.365341 17485 consensus_queue.cc:1035] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:37.369494 17469 mvcc.cc:205] Tried to move back new op lower bound from 7298809435602968576 to 7298809435504889856. Current Snapshot: MvccSnapshot[applied={T|T < 7298809435602968576}]
I20260620 06:11:37.393798 16896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:37.393937 16896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:37.395820 16896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Sending DeleteTablet for 3 replicas of tablet dc388347d50d4b01af1018bbe8c76cf0
I20260620 06:11:37.396317 17030 tablet_service.cc:1514] Processing DeleteTablet for tablet dc388347d50d4b01af1018bbe8c76cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:37 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:48138
I20260620 06:11:37.396412 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet dc388347d50d4b01af1018bbe8c76cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:37 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:37406
I20260620 06:11:37.396395 17179 tablet_service.cc:1514] Processing DeleteTablet for tablet dc388347d50d4b01af1018bbe8c76cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:37 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:55050
I20260620 06:11:37.396584 17490 tablet_replica.cc:330] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: stopping tablet replica
I20260620 06:11:37.396585 17492 tablet_replica.cc:330] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: stopping tablet replica
I20260620 06:11:37.396668 17490 raft_consensus.cc:2229] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:37.396688 17492 raft_consensus.cc:2229] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:37.396785 17490 raft_consensus.cc:2258] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:37.396787 17492 raft_consensus.cc:2258] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:37.396962 17491 tablet_replica.cc:330] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: stopping tablet replica
I20260620 06:11:37.397078 17491 raft_consensus.cc:2229] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:37.397194 17491 raft_consensus.cc:2258] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:37.397464 17490 ts_tablet_manager.cc:1905] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:37.397789 17492 ts_tablet_manager.cc:1905] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:37.397882 17491 ts_tablet_manager.cc:1905] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:37.399930 17491 ts_tablet_manager.cc:1918] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:37.399976 17490 ts_tablet_manager.cc:1918] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:37.400020 17491 log.cc:1196] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/dc388347d50d4b01af1018bbe8c76cf0
I20260620 06:11:37.400060 17490 log.cc:1196] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/dc388347d50d4b01af1018bbe8c76cf0
I20260620 06:11:37.400168 17492 ts_tablet_manager.cc:1918] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:11:37.400249 17492 log.cc:1196] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/dc388347d50d4b01af1018bbe8c76cf0
I20260620 06:11:37.400296 17491 ts_tablet_manager.cc:1939] T dc388347d50d4b01af1018bbe8c76cf0 P 572b5470ace247a484b59e0783c54aad: Deleting consensus metadata
I20260620 06:11:37.400341 17490 ts_tablet_manager.cc:1939] T dc388347d50d4b01af1018bbe8c76cf0 P 42c34bc9fc3b430b920663c5b46fe34c: Deleting consensus metadata
I20260620 06:11:37.400545 17492 ts_tablet_manager.cc:1939] T dc388347d50d4b01af1018bbe8c76cf0 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting consensus metadata
I20260620 06:11:37.400974 16887 catalog_manager.cc:4841] TS 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081): tablet dc388347d50d4b01af1018bbe8c76cf0 (table test-table [id=63e442c62f3348fb8f1e785f4f6de536]) successfully deleted
I20260620 06:11:37.401079 16886 catalog_manager.cc:4841] TS 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051): tablet dc388347d50d4b01af1018bbe8c76cf0 (table test-table [id=63e442c62f3348fb8f1e785f4f6de536]) successfully deleted
I20260620 06:11:37.401161 16889 catalog_manager.cc:4841] TS 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759): tablet dc388347d50d4b01af1018bbe8c76cf0 (table test-table [id=63e442c62f3348fb8f1e785f4f6de536]) successfully deleted
I20260620 06:11:37.401424 16896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:37.401809 16896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:37.405763 17030 tablet_service.cc:1467] Processing CreateTablet for tablet 4448bc536c6c4b3d8ad0a21d70354a30 (DEFAULT_TABLE table=test-table [id=a30020a9fa094fa592d30d63b9692877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:37.405799 17179 tablet_service.cc:1467] Processing CreateTablet for tablet 4448bc536c6c4b3d8ad0a21d70354a30 (DEFAULT_TABLE table=test-table [id=a30020a9fa094fa592d30d63b9692877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:37.405942 17319 tablet_service.cc:1467] Processing CreateTablet for tablet 4448bc536c6c4b3d8ad0a21d70354a30 (DEFAULT_TABLE table=test-table [id=a30020a9fa094fa592d30d63b9692877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:37.406013 17030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448bc536c6c4b3d8ad0a21d70354a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:37.406057 17179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448bc536c6c4b3d8ad0a21d70354a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:37.406154 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448bc536c6c4b3d8ad0a21d70354a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:37.407778 17501 tablet_bootstrap.cc:492] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Bootstrap starting.
I20260620 06:11:37.408128 17503 tablet_bootstrap.cc:492] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Bootstrap starting.
I20260620 06:11:37.408552 17501 tablet_bootstrap.cc:654] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:37.408918 17503 tablet_bootstrap.cc:654] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:37.409487 17502 tablet_bootstrap.cc:492] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c: Bootstrap starting.
I20260620 06:11:37.409613 17501 tablet_bootstrap.cc:492] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: No bootstrap required, opened a new log
I20260620 06:11:37.409678 17501 ts_tablet_manager.cc:1397] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:37.409894 17503 tablet_bootstrap.cc:492] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: No bootstrap required, opened a new log
I20260620 06:11:37.409950 17503 ts_tablet_manager.cc:1397] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:11:37.410030 17501 raft_consensus.cc:348] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.410127 17501 raft_consensus.cc:374] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:37.410167 17501 raft_consensus.cc:729] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572b5470ace247a484b59e0783c54aad, State: Initialized, Role: FOLLOWER
I20260620 06:11:37.410239 17502 tablet_bootstrap.cc:654] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:37.410234 17503 raft_consensus.cc:348] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.410346 17503 raft_consensus.cc:374] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:37.410301 17501 consensus_queue.cc:260] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.410390 17503 raft_consensus.cc:729] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f51b8b43684ecdbf26fb2ea7e538c7, State: Initialized, Role: FOLLOWER
I20260620 06:11:37.410514 17503 consensus_queue.cc:260] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.410564 17501 ts_tablet_manager.cc:1428] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:37.410742 17503 ts_tablet_manager.cc:1428] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:11:37.411877 17502 tablet_bootstrap.cc:492] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c: No bootstrap required, opened a new log
I20260620 06:11:37.411949 17502 ts_tablet_manager.cc:1397] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:37.412259 17502 raft_consensus.cc:348] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.412367 17502 raft_consensus.cc:374] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:37.412405 17502 raft_consensus.cc:729] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c34bc9fc3b430b920663c5b46fe34c, State: Initialized, Role: FOLLOWER
I20260620 06:11:37.412532 17502 consensus_queue.cc:260] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.412781 17502 ts_tablet_manager.cc:1428] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:37.419243 17507 raft_consensus.cc:482] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:37.419333 17507 raft_consensus.cc:504] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.420322 17507 leader_election.cc:288] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759), 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081)
I20260620 06:11:37.424350 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.420642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:57781 (local address 127.14.209.129:41759)
0620 06:11:37.420761 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.420765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.421052 (+ 287us) server_negotiation.cc:407] Connection header received
0620 06:11:37.421207 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.421212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.421261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.421327 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:37.421648 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.422171 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.423308 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.423737 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.423858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.423919 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.424175 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.424238 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:11:37.424367 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.420580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:50663 (local address 127.14.209.130:33081)
0620 06:11:37.420823 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.420837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.420870 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:37.420992 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.420998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.421049 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.421162 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:37.421547 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.422108 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.423362 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.423747 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.423879 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.423935 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.424184 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.424253 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260620 06:11:37.424381 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.420517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41759 (local address 127.14.209.131:57781)
0620 06:11:37.420917 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:37.420966 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:37.421068 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:37.421423 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:37.421425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:37.421439 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:37.421559 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.421565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.422295 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.422298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.423200 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.423209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.423872 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.423876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.424043 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.424072 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:11:37.424160 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:11:37.424229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":75,"thread_start_us":62,"threads_started":1}
I20260620 06:11:37.424386 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.420506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.14.209.131:50663)
0620 06:11:37.420779 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:37.420794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:37.420890 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:37.421265 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:37.421268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:37.421287 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:37.421423 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.421429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.422216 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.422221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.423218 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.423227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.423873 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.423877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.424040 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.424071 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 06:11:37.424155 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:11:37.424224 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":67,"thread_start_us":93,"threads_started":1}
I20260620 06:11:37.424685 17059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448bc536c6c4b3d8ad0a21d70354a30" candidate_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c34bc9fc3b430b920663c5b46fe34c" is_pre_election: true
I20260620 06:11:37.424798 17059 raft_consensus.cc:2454] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f51b8b43684ecdbf26fb2ea7e538c7 in term 0.
I20260620 06:11:37.424966 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448bc536c6c4b3d8ad0a21d70354a30" candidate_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572b5470ace247a484b59e0783c54aad" is_pre_election: true
I20260620 06:11:37.425104 17199 raft_consensus.cc:2454] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f51b8b43684ecdbf26fb2ea7e538c7 in term 0.
I20260620 06:11:37.425215 17273 leader_election.cc:302] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:37.425446 17507 raft_consensus.cc:2781] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:37.425529 17507 raft_consensus.cc:482] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:37.425581 17507 raft_consensus.cc:3037] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:37.426622 17507 raft_consensus.cc:504] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.426935 17507 leader_election.cc:288] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [CANDIDATE]: Term 1 election: Requested vote from peers 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759), 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081)
I20260620 06:11:37.427258 17059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448bc536c6c4b3d8ad0a21d70354a30" candidate_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c34bc9fc3b430b920663c5b46fe34c"
I20260620 06:11:37.427373 17059 raft_consensus.cc:3037] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:37.428231 17059 raft_consensus.cc:2454] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f51b8b43684ecdbf26fb2ea7e538c7 in term 1.
I20260620 06:11:37.428481 17199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448bc536c6c4b3d8ad0a21d70354a30" candidate_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572b5470ace247a484b59e0783c54aad"
I20260620 06:11:37.428524 17273 leader_election.cc:302] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c34bc9fc3b430b920663c5b46fe34c, 90f51b8b43684ecdbf26fb2ea7e538c7; no voters:
I20260620 06:11:37.428599 17199 raft_consensus.cc:3037] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:37.428690 17507 raft_consensus.cc:2781] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:37.428892 17507 raft_consensus.cc:686] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 LEADER]: Becoming Leader. State: Replica: 90f51b8b43684ecdbf26fb2ea7e538c7, State: Running, Role: LEADER
I20260620 06:11:37.429104 17507 consensus_queue.cc:237] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } }
I20260620 06:11:37.429668 17199 raft_consensus.cc:2454] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f51b8b43684ecdbf26fb2ea7e538c7 in term 1.
I20260620 06:11:37.431499 16898 catalog_manager.cc:5495] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f51b8b43684ecdbf26fb2ea7e538c7" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45051 } health_report { overall_health: HEALTHY } } }
I20260620 06:11:37.449643 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.446048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.14.209.131:45051)
0620 06:11:37.446354 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.446363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.446374 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:11:37.446447 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.446449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.446508 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.446641 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:37.446869 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.447490 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.448300 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.448554 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.448630 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.448825 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:37.449268 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:37.449372 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.449418 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.449459 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":119,"threads_started":1}
I20260620 06:11:37.458829 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.455854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:56505 (local address 127.14.209.130:33081)
0620 06:11:37.455974 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.455980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.456003 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:11:37.456143 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.456147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.456199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.456281 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:37.456509 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.456999 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.457831 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.458058 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.458167 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.458423 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:37.458492 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:37.458572 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.458656 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.458703 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:37.458859 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.455665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33081 (local address 127.14.209.131:56505)
0620 06:11:37.455933 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:37.455947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:37.456037 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:37.456291 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:37.456294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:37.456302 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:37.456408 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.456413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.457103 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.457108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.457716 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.457723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.458167 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.458171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.458263 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.458297 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:37.458577 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:37.458586 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:11:37.458648 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:11:37.458718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260620 06:11:37.463990 17199 raft_consensus.cc:1261] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Refusing update from remote peer 90f51b8b43684ecdbf26fb2ea7e538c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:37.464036 17059 raft_consensus.cc:1261] T 4448bc536c6c4b3d8ad0a21d70354a30 P 42c34bc9fc3b430b920663c5b46fe34c [term 1 FOLLOWER]: Refusing update from remote peer 90f51b8b43684ecdbf26fb2ea7e538c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:37.464455 17521 consensus_queue.cc:1035] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572b5470ace247a484b59e0783c54aad" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:37.464668 17507 consensus_queue.cc:1035] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42c34bc9fc3b430b920663c5b46fe34c" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:37.484700 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.480848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:56903 (local address 127.14.209.131:45051)
0620 06:11:37.480960 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:37.480964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:37.481159 (+ 195us) server_negotiation.cc:407] Connection header received
0620 06:11:37.481265 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:37.481267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:37.481309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:37.481389 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:37.481672 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.482415 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.483529 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.483862 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.483951 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.484262 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:37.484337 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:37.484397 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:37.484532 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 06:11:37.484583 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:11:37.485116 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:37.480802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45051 (local address 127.14.209.130:56903)
0620 06:11:37.481063 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:37.481078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:37.481182 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:37.481466 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:37.481470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:37.481478 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:11:37.481567 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.481571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.482552 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.482556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.483300 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:37.483310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:37.484025 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:37.484029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:37.484112 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:37.484140 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:11:37.484419 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:11:37.484425 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:11:37.484902 (+ 477us) client_negotiation.cc:240] Negotiation successful
0620 06:11:37.484982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":118,"threads_started":1}
W20260620 06:11:37.486202 17473 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:11:39.045822 16898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:39.045976 16898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:39.049049 16898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355f97f62b234102bd02525c40b87308: Sending DeleteTablet for 3 replicas of tablet 4448bc536c6c4b3d8ad0a21d70354a30
I20260620 06:11:39.049877 17030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448bc536c6c4b3d8ad0a21d70354a30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:39 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:48138
I20260620 06:11:39.049996 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448bc536c6c4b3d8ad0a21d70354a30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:39 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:37406
I20260620 06:11:39.050136 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16974
I20260620 06:11:39.050175 17179 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448bc536c6c4b3d8ad0a21d70354a30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:39 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:55050
I20260620 06:11:39.050841 17559 tablet_replica.cc:330] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: stopping tablet replica
I20260620 06:11:39.051018 17559 raft_consensus.cc:2229] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:39.051344 17559 raft_consensus.cc:2258] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:39.051993 17559 ts_tablet_manager.cc:1905] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:39.052346 17560 tablet_replica.cc:330] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: stopping tablet replica
I20260620 06:11:39.052469 17560 raft_consensus.cc:2229] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:39.052580 17560 raft_consensus.cc:2258] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:39.053148 17560 ts_tablet_manager.cc:1905] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:39.054522 17559 ts_tablet_manager.cc:1918] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:11:39.054615 17559 log.cc:1196] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/4448bc536c6c4b3d8ad0a21d70354a30
I20260620 06:11:39.054903 17559 ts_tablet_manager.cc:1939] T 4448bc536c6c4b3d8ad0a21d70354a30 P 90f51b8b43684ecdbf26fb2ea7e538c7: Deleting consensus metadata
I20260620 06:11:39.055687 17560 ts_tablet_manager.cc:1918] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:11:39.055727 16886 catalog_manager.cc:4841] TS 90f51b8b43684ecdbf26fb2ea7e538c7 (127.14.209.131:45051): tablet 4448bc536c6c4b3d8ad0a21d70354a30 (table test-table [id=a30020a9fa094fa592d30d63b9692877]) successfully deleted
I20260620 06:11:39.055774 17560 log.cc:1196] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/4448bc536c6c4b3d8ad0a21d70354a30
I20260620 06:11:39.056016 17560 ts_tablet_manager.cc:1939] T 4448bc536c6c4b3d8ad0a21d70354a30 P 572b5470ace247a484b59e0783c54aad: Deleting consensus metadata
I20260620 06:11:39.056614 16887 catalog_manager.cc:4841] TS 572b5470ace247a484b59e0783c54aad (127.14.209.130:33081): tablet 4448bc536c6c4b3d8ad0a21d70354a30 (table test-table [id=a30020a9fa094fa592d30d63b9692877]) successfully deleted
W20260620 06:11:39.060478 16889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:41759 (error 108)
I20260620 06:11:39.060789 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 17114
I20260620 06:11:39.062829 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:39.062391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41759 (local address 127.0.0.1:48162)
0620 06:11:39.062680 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:39.062742 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:41759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
W20260620 06:11:39.063010 16889 catalog_manager.cc:4584] TS 42c34bc9fc3b430b920663c5b46fe34c (127.14.209.129:41759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4448bc536c6c4b3d8ad0a21d70354a30: Network error: Client connection negotiation failed: client connection to 127.14.209.129:41759: connect: Connection refused (error 111)
I20260620 06:11:39.067703 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 17254
I20260620 06:11:39.074144 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 16870
2026-06-20T06:11:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6230 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:39 dist-test-slave-zt01 krb5kdc[17565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:39 dist-test-slave-zt01 krb5kdc[17565](info): set up 2 sockets
Jun 20 06:11:39 dist-test-slave-zt01 krb5kdc[17565](info): commencing operation
krb5kdc: starting...
W20260620 06:11:40.138988 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:40.257457 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:36643
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:40.360911 17578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:40.361181 17578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:40.361259 17578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:40.365146 17578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:40.365227 17578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:40.365265 17578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:40.365316 17578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:40.365353 17578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:40.370551 17578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:36643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:40.371512 17578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:40.372443 17578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:40.380599 17583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:40.380681 17578 server_base.cc:1034] running on GCE node
W20260620 06:11:40.380878 17586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:40.380946 17584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:40.381440 17578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:40.382463 17578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:40.383657 17578 hybrid_clock.cc:648] HybridClock initialized: now 1781935900383638 us; error 31 us; skew 500 ppm
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:40.386255 17578 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:40.387125 17578 webserver.cc:460] Webserver started at http://127.14.209.190:36711/ using document root <none> and password file <none>
I20260620 06:11:40.387651 17578 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:40.387743 17578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:40.387961 17578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:40.389869 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "a718ff86948144b99142a96bb8f1b500"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "f4f4d9288ac0d4f8f083a1954e94f38a"
server_key_iv: "53e5c25f7f77271077d66af9d029b6ea"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.390411 17578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "a718ff86948144b99142a96bb8f1b500"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "f4f4d9288ac0d4f8f083a1954e94f38a"
server_key_iv: "53e5c25f7f77271077d66af9d029b6ea"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.393941 17578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:40.396221 17593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.397404 17578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:40.397505 17578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "a718ff86948144b99142a96bb8f1b500"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "f4f4d9288ac0d4f8f083a1954e94f38a"
server_key_iv: "53e5c25f7f77271077d66af9d029b6ea"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.397583 17578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:40.409595 17578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:40.412446 17578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:40.412593 17578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:40.419165 17578 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:36643
I20260620 06:11:40.419174 17645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:36643 every 8 connection(s)
I20260620 06:11:40.420047 17578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:40.422605 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:40.422674 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 17578
I20260620 06:11:40.422781 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:40.423046 15174 external_mini_cluster.cc:1428] Setting key dedef302a0eafed2daa98bbf64bed9a0
I20260620 06:11:40.428467 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500: Bootstrap starting.
I20260620 06:11:40.430397 17646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:40.431139 17646 log.cc:824] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:40.432905 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500: No bootstrap required, opened a new log
I20260620 06:11:40.437078 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:40.424430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.14.209.190:36643)
0620 06:11:40.424829 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:40.424846 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:40.424879 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:40.425631 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:40.425647 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:40.425934 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:40.426246 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:40.426999 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.427986 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.428865 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.429308 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.429476 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:40.432150 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:40.432155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:40.432166 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:40.432189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:40.434076 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:40.434478 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:40.434484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:40.434493 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:40.434622 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:40.434900 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:40.434905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:40.434908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:40.435496 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:40.435680 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:40.435933 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 06:11:40.436199 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260620 06:11:40.437342 17646 raft_consensus.cc:348] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a718ff86948144b99142a96bb8f1b500" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36643 } }
I20260620 06:11:40.437579 17646 raft_consensus.cc:374] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:40.437637 17646 raft_consensus.cc:729] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a718ff86948144b99142a96bb8f1b500, State: Initialized, Role: FOLLOWER
I20260620 06:11:40.438058 17646 consensus_queue.cc:260] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a718ff86948144b99142a96bb8f1b500" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36643 } }
I20260620 06:11:40.438189 17646 raft_consensus.cc:388] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:40.438265 17646 raft_consensus.cc:482] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:40.438359 17646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:40.439314 17646 raft_consensus.cc:504] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a718ff86948144b99142a96bb8f1b500" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36643 } }
I20260620 06:11:40.439625 17646 leader_election.cc:302] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a718ff86948144b99142a96bb8f1b500; no voters:
I20260620 06:11:40.439915 17646 leader_election.cc:288] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:40.440013 17651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:40.440240 17651 raft_consensus.cc:686] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [term 1 LEADER]: Becoming Leader. State: Replica: a718ff86948144b99142a96bb8f1b500, State: Running, Role: LEADER
I20260620 06:11:40.440560 17651 consensus_queue.cc:237] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a718ff86948144b99142a96bb8f1b500" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36643 } }
I20260620 06:11:40.440991 17646 sys_catalog.cc:564] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:40.442226 17652 sys_catalog.cc:455] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a718ff86948144b99142a96bb8f1b500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a718ff86948144b99142a96bb8f1b500" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36643 } } }
I20260620 06:11:40.442219 17653 sys_catalog.cc:455] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a718ff86948144b99142a96bb8f1b500. Latest consensus state: current_term: 1 leader_uuid: "a718ff86948144b99142a96bb8f1b500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a718ff86948144b99142a96bb8f1b500" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 36643 } } }
I20260620 06:11:40.442363 17652 sys_catalog.cc:458] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:40.442369 17653 sys_catalog.cc:458] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500 [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:40.445698 17666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:40.445788 17666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:40.445899 17662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:40.446601 17662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:40.452075 17662 catalog_manager.cc:1347] Generated new cluster ID: 21d6c0c1036b4f7e92ac2f0fc3dbdda2
I20260620 06:11:40.452147 17662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:40.473393 17662 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:40.474247 17662 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:40.482437 17662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a718ff86948144b99142a96bb8f1b500: Generated new TSK 0
I20260620 06:11:40.483115 17662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:40.518676 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36643
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:40.623783 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:40.624056 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:40.624125 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:40.628034 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:40.628113 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:40.628211 17672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:40.633124 17672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:40.634056 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:40.634969 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:40.644129 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:40.644145 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:40.644207 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:40.644196 17672 server_base.cc:1034] running on GCE node
I20260620 06:11:40.644692 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:40.645316 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:40.646485 17672 hybrid_clock.cc:648] HybridClock initialized: now 1781935900646458 us; error 37 us; skew 500 ppm
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:40.649113 17672 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:40.649979 17672 webserver.cc:460] Webserver started at http://127.14.209.129:36699/ using document root <none> and password file <none>
I20260620 06:11:40.650440 17672 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:40.650496 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:40.650640 17672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:40.652518 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "d8009563cb9e491faa4d6c22f8a1dbab"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "1577c9780a808e06ee064d52a3a29740"
server_key_iv: "bc92dc2af3fcf2224ba5e05e685147f8"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.653013 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "d8009563cb9e491faa4d6c22f8a1dbab"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "1577c9780a808e06ee064d52a3a29740"
server_key_iv: "bc92dc2af3fcf2224ba5e05e685147f8"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.656348 17672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:40.658475 17687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.659700 17672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:40.659832 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "d8009563cb9e491faa4d6c22f8a1dbab"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "1577c9780a808e06ee064d52a3a29740"
server_key_iv: "bc92dc2af3fcf2224ba5e05e685147f8"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.659942 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:40.674846 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:40.677654 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:40.677934 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:40.678524 17672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:40.679308 17672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:40.679384 17672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.679450 17672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:40.679512 17672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.688520 17672 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:41265
I20260620 06:11:40.688552 17800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:41265 every 8 connection(s)
I20260620 06:11:40.689298 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:40.694886 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 17672
I20260620 06:11:40.694991 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:40.695277 15174 external_mini_cluster.cc:1428] Setting key 3f5de35220aaa42cc42c67788988bd6a
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:40.704305 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:40.690941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:41563 (local address 127.14.209.190:36643)
0620 06:11:40.691109 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:40.691114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:40.691885 (+ 771us) server_negotiation.cc:407] Connection header received
0620 06:11:40.693090 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:40.693093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:40.693150 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:40.693255 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:40.694335 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.695034 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.696536 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.696829 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.696981 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:40.699667 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:40.699673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:40.699678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:40.699710 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:40.701705 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:40.702419 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:40.702425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:40.702446 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:40.702543 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:40.703110 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:40.703114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:40.703116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:40.703393 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:40.703511 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:40.703989 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 06:11:40.704119 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:40.704959 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:40.691327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:36643 (local address 127.14.209.129:41563)
0620 06:11:40.691750 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:40.691786 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:40.692901 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:40.693414 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:40.693420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:40.693726 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:40.694189 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:40.694202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.695193 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.695196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:40.696384 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:40.696394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.697019 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.697022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:40.697158 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:40.697780 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:40.697795 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:40.699485 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:40.701884 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:40.701893 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:40.701928 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:40.702271 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:40.702680 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:40.702685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:40.702688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:40.702949 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:40.703531 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:40.703536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:40.703752 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 06:11:40.703905 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:11:40.704141 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:40.706128 17801 heartbeater.cc:347] Connected to a master server at 127.14.209.190:36643
I20260620 06:11:40.706323 17801 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:40.706779 17801 heartbeater.cc:511] Master 127.14.209.190:36643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:11:40.708385 17610 ts_manager.cc:194] Registered new tserver with Master: d8009563cb9e491faa4d6c22f8a1dbab (127.14.209.129:41265)
I20260620 06:11:40.709786 17610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:41563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:40.722015 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36643
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:40.826576 17806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:40.826789 17806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:40.826828 17806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:40.830731 17806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:40.830778 17806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:40.830870 17806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:40.835817 17806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:40.836741 17806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:40.837668 17806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:11:40.847575 17806 server_base.cc:1034] running on GCE node
W20260620 06:11:40.847620 17814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:40.847772 17812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:40.847573 17811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:40.848179 17806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:40.848788 17806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:40.850013 17806 hybrid_clock.cc:648] HybridClock initialized: now 1781935900849984 us; error 41 us; skew 500 ppm
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:40.852707 17806 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:40.853585 17806 webserver.cc:460] Webserver started at http://127.14.209.130:33687/ using document root <none> and password file <none>
I20260620 06:11:40.854110 17806 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:40.854171 17806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:40.854382 17806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:40.856592 17806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "dd91db0112c6498e9f7ab3818fc45094"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "9bcda10d5dde001125aeebdc57ba80d3"
server_key_iv: "fdd1f18520f243d23c05fd8577f87f11"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.857131 17806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "dd91db0112c6498e9f7ab3818fc45094"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "9bcda10d5dde001125aeebdc57ba80d3"
server_key_iv: "fdd1f18520f243d23c05fd8577f87f11"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.860591 17806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 06:11:40.862869 17821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.864135 17806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:40.864248 17806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "dd91db0112c6498e9f7ab3818fc45094"
format_stamp: "Formatted at 2026-06-20 06:11:40 on dist-test-slave-zt01"
server_key: "9bcda10d5dde001125aeebdc57ba80d3"
server_key_iv: "fdd1f18520f243d23c05fd8577f87f11"
server_key_version: "encryptionkey@0"
I20260620 06:11:40.864346 17806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:40.883900 17806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:40.886493 17806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:40.886682 17806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:40.887292 17806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:40.888052 17806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:40.888098 17806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.888164 17806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:40.888207 17806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:40.896816 17806 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:41585
I20260620 06:11:40.896854 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:41585 every 8 connection(s)
I20260620 06:11:40.897617 17806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
Jun 20 06:11:40 dist-test-slave-zt01 krb5kdc[17565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:40.907542 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 17806
I20260620 06:11:40.907655 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:40.908056 15174 external_mini_cluster.cc:1428] Setting key b1e78b2777f42a3b0f84c1f67d90aaf9
I20260620 06:11:40.910511 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:40.899263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:41303 (local address 127.14.209.190:36643)
0620 06:11:40.899388 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:40.899391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:40.900152 (+ 761us) server_negotiation.cc:407] Connection header received
0620 06:11:40.901397 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:40.901400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:40.901455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:40.901564 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:40.902624 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.903213 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.904313 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.904542 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.904661 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:40.906941 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:40.906944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:40.906946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:40.906970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:40.908498 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:40.908983 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:40.908986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:40.908988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:40.909069 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:40.909457 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:40.909461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:40.909463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:40.909642 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:40.909750 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:40.910226 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 06:11:40.910341 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:40.911332 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:40.899555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:36643 (local address 127.14.209.130:41303)
0620 06:11:40.899977 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:40.900016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:40.901198 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:40.901719 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:40.901725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:40.902041 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:40.902478 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:40.902490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.903331 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.903335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:40.904188 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:40.904197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:40.904685 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:40.904689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:40.904859 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:40.905351 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:40.905366 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:40.906755 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:40.908635 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:40.908639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:40.908648 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:40.908882 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:40.909179 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:40.909183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:40.909184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:40.909353 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:40.909757 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:40.909765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:40.910006 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 06:11:40.910184 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 06:11:40.910467 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260620 06:11:40.912405 17935 heartbeater.cc:347] Connected to a master server at 127.14.209.190:36643
I20260620 06:11:40.912633 17935 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:40.913075 17935 heartbeater.cc:511] Master 127.14.209.190:36643 requested a full tablet report, sending...
I20260620 06:11:40.914086 17610 ts_manager.cc:194] Registered new tserver with Master: dd91db0112c6498e9f7ab3818fc45094 (127.14.209.130:41585)
I20260620 06:11:40.914990 17610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:41303
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:40.934324 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36643
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:41.037927 17940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:41.038141 17940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:41.038179 17940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:41.042229 17940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:41.042309 17940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:41.042368 17940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:41.047528 17940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:41.048348 17940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:41.049258 17940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:41.058138 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:41.058130 17946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:41.058089 17945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:41.058349 17940 server_base.cc:1034] running on GCE node
I20260620 06:11:41.058838 17940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:41.059496 17940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:41.060701 17940 hybrid_clock.cc:648] HybridClock initialized: now 1781935901060660 us; error 64 us; skew 500 ppm
Jun 20 06:11:41 dist-test-slave-zt01 krb5kdc[17565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:41.063370 17940 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:41.064252 17940 webserver.cc:460] Webserver started at http://127.14.209.131:36157/ using document root <none> and password file <none>
I20260620 06:11:41.064772 17940 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:41.064859 17940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:41.065075 17940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:41.066885 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "7da579cbca374b6c9a1e2ae37db55c1a"
format_stamp: "Formatted at 2026-06-20 06:11:41 on dist-test-slave-zt01"
server_key: "3eff788486a78f30f7e261ef2e8ddce6"
server_key_iv: "b8531c99ea530e268d834790f854f92a"
server_key_version: "encryptionkey@0"
I20260620 06:11:41.067461 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "7da579cbca374b6c9a1e2ae37db55c1a"
format_stamp: "Formatted at 2026-06-20 06:11:41 on dist-test-slave-zt01"
server_key: "3eff788486a78f30f7e261ef2e8ddce6"
server_key_iv: "b8531c99ea530e268d834790f854f92a"
server_key_version: "encryptionkey@0"
I20260620 06:11:41.071025 17940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:41.073335 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:41.074393 17940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:11:41.074529 17940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "7da579cbca374b6c9a1e2ae37db55c1a"
format_stamp: "Formatted at 2026-06-20 06:11:41 on dist-test-slave-zt01"
server_key: "3eff788486a78f30f7e261ef2e8ddce6"
server_key_iv: "b8531c99ea530e268d834790f854f92a"
server_key_version: "encryptionkey@0"
I20260620 06:11:41.074632 17940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:41.087760 17940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:41.091557 17940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:41.091753 17940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:41.092332 17940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:41.093122 17940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:41.093192 17940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:41.093262 17940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:41.093305 17940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:41.102345 17940 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:44111
I20260620 06:11:41.102367 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:44111 every 8 connection(s)
I20260620 06:11:41.103256 17940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:41.109611 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 17940
I20260620 06:11:41.109737 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:41.110036 15174 external_mini_cluster.cc:1428] Setting key 14d552aeac8da51addc84bc504a7f6cc
Jun 20 06:11:41 dist-test-slave-zt01 krb5kdc[17565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:41.116631 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:41.104943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:53885 (local address 127.14.209.190:36643)
0620 06:11:41.105086 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:41.105090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:41.105815 (+ 725us) server_negotiation.cc:407] Connection header received
0620 06:11:41.107117 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:41.107121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:41.107177 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:41.107275 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:41.108341 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:41.108864 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:41.110072 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:41.110335 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:41.110444 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:41.113069 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:41.113073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:41.113075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:41.113096 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:41.114910 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:41.115434 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:41.115436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:41.115438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:41.115521 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:41.115791 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:41.115793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:41.115794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:41.115940 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:41.116020 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:41.116422 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 06:11:41.116522 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:11:41.117338 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:41.105283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:36643 (local address 127.14.209.131:53885)
0620 06:11:41.105684 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:41.105722 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:41.106863 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:41.107433 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:41.107439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:41.107767 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:41.108201 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:41.108213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:41.108995 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:41.108998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:41.109950 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:41.109959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:41.110475 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:41.110479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:41.110693 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:41.111345 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:41.111365 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:41.112909 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:41.115013 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:41.115023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:41.115035 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:41.115349 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:41.115601 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:41.115603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:41.115604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:41.115721 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:41.116028 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:41.116032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:41.116231 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 06:11:41.116374 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 06:11:41.116596 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260620 06:11:41.118343 18069 heartbeater.cc:347] Connected to a master server at 127.14.209.190:36643
I20260620 06:11:41.118569 18069 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:41.119025 18069 heartbeater.cc:511] Master 127.14.209.190:36643 requested a full tablet report, sending...
I20260620 06:11:41.119987 17610 ts_manager.cc:194] Registered new tserver with Master: 7da579cbca374b6c9a1e2ae37db55c1a (127.14.209.131:44111)
I20260620 06:11:41.120528 17610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:53885
I20260620 06:11:41.123440 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:11:41.135001 17649 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:41.133738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.14.209.190:36643)
0620 06:11:41.133882 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:41.133889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:41.134029 (+ 140us) server_negotiation.cc:407] Connection header received
0620 06:11:41.134124 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:41.134126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:41.134161 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:41.134225 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:41.134876 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41942: BlockingRecv error: recv got EOF from 127.0.0.1:41942 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:11:41.135882 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 17672
I20260620 06:11:41.141317 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 17806
I20260620 06:11:41.146427 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 17940
I20260620 06:11:41.151401 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 17578
2026-06-20T06:11:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2076 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:41 dist-test-slave-zt01 krb5kdc[18081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:41 dist-test-slave-zt01 krb5kdc[18081](info): set up 2 sockets
Jun 20 06:11:41 dist-test-slave-zt01 krb5kdc[18081](info): commencing operation
krb5kdc: starting...
W20260620 06:11:42.211527 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:42 dist-test-slave-zt01 krb5kdc[18081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:42.336813 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40215
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:42.446753 18094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:42.447016 18094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:42.447141 18094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:42.451023 18094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:42.451124 18094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:42.451161 18094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:42.451220 18094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:42.451259 18094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:42.456350 18094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:42.457291 18094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:42.458221 18094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:42.466151 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:42.466157 18100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:42.466166 18102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:42.466213 18094 server_base.cc:1034] running on GCE node
I20260620 06:11:42.466861 18094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:42.467895 18094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:42.469095 18094 hybrid_clock.cc:648] HybridClock initialized: now 1781935902469077 us; error 29 us; skew 500 ppm
Jun 20 06:11:42 dist-test-slave-zt01 krb5kdc[18081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935902, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:42.475842 18094 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:42.476707 18094 webserver.cc:460] Webserver started at http://127.14.209.190:39937/ using document root <none> and password file <none>
I20260620 06:11:42.477226 18094 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:42.477314 18094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:42.477557 18094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:42.479620 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "488c494daf1a4dd899b20418234975ac"
format_stamp: "Formatted at 2026-06-20 06:11:42 on dist-test-slave-zt01"
server_key: "9c06bc698052df480c9eda483ab9253f"
server_key_iv: "9d85c4d3f34aef8de43c7d1f8c78065a"
server_key_version: "encryptionkey@0"
I20260620 06:11:42.480151 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "488c494daf1a4dd899b20418234975ac"
format_stamp: "Formatted at 2026-06-20 06:11:42 on dist-test-slave-zt01"
server_key: "9c06bc698052df480c9eda483ab9253f"
server_key_iv: "9d85c4d3f34aef8de43c7d1f8c78065a"
server_key_version: "encryptionkey@0"
I20260620 06:11:42.483678 18094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:42.485863 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:42.486972 18094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:42.487151 18094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "488c494daf1a4dd899b20418234975ac"
format_stamp: "Formatted at 2026-06-20 06:11:42 on dist-test-slave-zt01"
server_key: "9c06bc698052df480c9eda483ab9253f"
server_key_iv: "9d85c4d3f34aef8de43c7d1f8c78065a"
server_key_version: "encryptionkey@0"
I20260620 06:11:42.487260 18094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:42.506072 18094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:42.508818 18094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:42.509047 18094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:42.516373 18094 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40215
I20260620 06:11:42.516448 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40215 every 8 connection(s)
I20260620 06:11:42.517288 18094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:42.521028 18162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:42.522336 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18094
I20260620 06:11:42.522477 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:42.522975 15174 external_mini_cluster.cc:1428] Setting key b62c9643aa78f56226b4f06210930f15
I20260620 06:11:42.527428 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac: Bootstrap starting.
I20260620 06:11:42.529500 18162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:42.530311 18162 log.cc:824] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:11:42 dist-test-slave-zt01 krb5kdc[18081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:42.532222 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac: No bootstrap required, opened a new log
I20260620 06:11:42.535683 18162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488c494daf1a4dd899b20418234975ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40215 } }
I20260620 06:11:42.535962 18162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:42.536051 18162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488c494daf1a4dd899b20418234975ac, State: Initialized, Role: FOLLOWER
I20260620 06:11:42.536441 18162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488c494daf1a4dd899b20418234975ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40215 } }
I20260620 06:11:42.536553 18162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:42.536634 18162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:42.536733 18162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:42.536914 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:42.524416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.14.209.190:40215)
0620 06:11:42.524856 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:42.524871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:42.524904 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:42.525603 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:42.525618 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:42.525920 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:42.526292 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:42.527112 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:42.528105 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:42.528942 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:42.529396 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:42.529573 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:42.532219 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:42.532225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:42.532236 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:42.532280 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:42.533973 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:42.534396 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:42.534401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:42.534407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:42.534516 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:42.534792 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:42.534795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:42.534797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:42.535228 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:42.535432 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:42.535684 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 06:11:42.535987 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":127,"threads_started":1}
I20260620 06:11:42.537571 18162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488c494daf1a4dd899b20418234975ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40215 } }
I20260620 06:11:42.537945 18162 leader_election.cc:302] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 488c494daf1a4dd899b20418234975ac; no voters:
I20260620 06:11:42.538231 18162 leader_election.cc:288] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:42.538341 18167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:42.538548 18167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [term 1 LEADER]: Becoming Leader. State: Replica: 488c494daf1a4dd899b20418234975ac, State: Running, Role: LEADER
I20260620 06:11:42.538834 18167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488c494daf1a4dd899b20418234975ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40215 } }
I20260620 06:11:42.539422 18162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:42.539638 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:42.541929 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 488c494daf1a4dd899b20418234975ac. Latest consensus state: current_term: 1 leader_uuid: "488c494daf1a4dd899b20418234975ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488c494daf1a4dd899b20418234975ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40215 } } }
I20260620 06:11:42.542047 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "488c494daf1a4dd899b20418234975ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488c494daf1a4dd899b20418234975ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40215 } } }
I20260620 06:11:42.542146 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:42.542145 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:42.544378 18183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:42.544518 18183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:42.544620 18184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:42.545540 18184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:42.551282 18184 catalog_manager.cc:1347] Generated new cluster ID: b9026d2a9a614413916a61ea0269b6ed
I20260620 06:11:42.551338 18184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:42.557025 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:42.552649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.14.209.190:40215)
0620 06:11:42.552759 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:42.552764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:42.553002 (+ 238us) server_negotiation.cc:407] Connection header received
0620 06:11:42.553141 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:42.553144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:42.553191 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:42.553276 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:42.553735 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:42.554252 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:42.555099 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:42.555342 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:42.555450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:42.556232 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:42.556236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:42.556239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:42.556619 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:42.556644 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:42.556658 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:42.556756 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:42.556837 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 06:11:42.556893 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:42.562827 18184 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:42.563938 18184 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:42.570830 18184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 488c494daf1a4dd899b20418234975ac: Generated new TSK 0
I20260620 06:11:42.571431 18184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:42.578127 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18094
2026-06-20T06:11:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1423 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:42 dist-test-slave-zt01 krb5kdc[18195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:42 dist-test-slave-zt01 krb5kdc[18195](info): set up 2 sockets
Jun 20 06:11:42 dist-test-slave-zt01 krb5kdc[18195](info): commencing operation
krb5kdc: starting...
W20260620 06:11:43.618954 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:43 dist-test-slave-zt01 krb5kdc[18195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:43.737931 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:44209
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:44209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:43.842795 18208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:43.843030 18208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:43.843083 18208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:43.846962 18208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:43.847019 18208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:43.847038 18208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:43.847070 18208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:43.847085 18208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:43.852097 18208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:44209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:44209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:43.853423 18208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:43.854425 18208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:11:43.862608 18208 server_base.cc:1034] running on GCE node
W20260620 06:11:43.862589 18213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:43.862567 18214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:43.862799 18216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:43.863401 18208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:43.864427 18208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:43.865612 18208 hybrid_clock.cc:648] HybridClock initialized: now 1781935903865595 us; error 29 us; skew 500 ppm
Jun 20 06:11:43 dist-test-slave-zt01 krb5kdc[18195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935903, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:43.868261 18208 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:43.869000 18208 webserver.cc:460] Webserver started at http://127.14.209.190:43507/ using document root <none> and password file <none>
I20260620 06:11:43.869472 18208 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:43.869527 18208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:43.869676 18208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:43.871567 18208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "e575ee0c8c144974a900494274ac7ce1"
format_stamp: "Formatted at 2026-06-20 06:11:43 on dist-test-slave-zt01"
server_key: "b5d230fa16066f6003db5856e2d71951"
server_key_iv: "42a2be03a53a4e544f0ad704f3b87c68"
server_key_version: "encryptionkey@0"
I20260620 06:11:43.872191 18208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "e575ee0c8c144974a900494274ac7ce1"
format_stamp: "Formatted at 2026-06-20 06:11:43 on dist-test-slave-zt01"
server_key: "b5d230fa16066f6003db5856e2d71951"
server_key_iv: "42a2be03a53a4e544f0ad704f3b87c68"
server_key_version: "encryptionkey@0"
I20260620 06:11:43.875973 18208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:43.878337 18223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:43.879572 18208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:43.879693 18208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "e575ee0c8c144974a900494274ac7ce1"
format_stamp: "Formatted at 2026-06-20 06:11:43 on dist-test-slave-zt01"
server_key: "b5d230fa16066f6003db5856e2d71951"
server_key_iv: "42a2be03a53a4e544f0ad704f3b87c68"
server_key_version: "encryptionkey@0"
I20260620 06:11:43.879787 18208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:43.897531 18208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:43.900646 18208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:43.900838 18208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:43.907548 18208 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:44209
I20260620 06:11:43.907869 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:44209 every 8 connection(s)
I20260620 06:11:43.908456 18208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:43.911353 18276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:43.913231 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18208
I20260620 06:11:43.913352 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:43.913645 15174 external_mini_cluster.cc:1428] Setting key 9ff81ad03c2c454a29f1727cc8fd337b
I20260620 06:11:43.918012 18276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1: Bootstrap starting.
I20260620 06:11:43.920405 18276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:43.921166 18276 log.cc:824] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:11:43 dist-test-slave-zt01 krb5kdc[18195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:43.923151 18276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1: No bootstrap required, opened a new log
I20260620 06:11:43.927796 18276 raft_consensus.cc:348] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e575ee0c8c144974a900494274ac7ce1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44209 } }
I20260620 06:11:43.928061 18276 raft_consensus.cc:374] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:43.928114 18276 raft_consensus.cc:729] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e575ee0c8c144974a900494274ac7ce1, State: Initialized, Role: FOLLOWER
I20260620 06:11:43.928514 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:43.914956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.14.209.190:44209)
0620 06:11:43.915385 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:43.915407 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:43.915456 (+ 49us) server_negotiation.cc:407] Connection header received
0620 06:11:43.916259 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:43.916281 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:43.916663 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:43.917010 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:43.917664 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:43.919083 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:43.919888 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:43.920593 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:43.920756 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:43.923455 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:43.923463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:43.923480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:43.923511 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:43.925421 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:43.925867 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:43.925875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:43.925881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:43.926016 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:43.926312 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:43.926315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:43.926317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:43.926783 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:43.926995 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:43.927325 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 06:11:43.927594 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":157,"threads_started":1}
I20260620 06:11:43.928556 18276 consensus_queue.cc:260] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e575ee0c8c144974a900494274ac7ce1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44209 } }
I20260620 06:11:43.928697 18276 raft_consensus.cc:388] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:43.928762 18276 raft_consensus.cc:482] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:43.928854 18276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:43.929872 18276 raft_consensus.cc:504] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e575ee0c8c144974a900494274ac7ce1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44209 } }
I20260620 06:11:43.930197 18276 leader_election.cc:302] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e575ee0c8c144974a900494274ac7ce1; no voters:
I20260620 06:11:43.930526 18276 leader_election.cc:288] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:43.930641 18281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:43.930929 18281 raft_consensus.cc:686] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [term 1 LEADER]: Becoming Leader. State: Replica: e575ee0c8c144974a900494274ac7ce1, State: Running, Role: LEADER
I20260620 06:11:43.931259 18281 consensus_queue.cc:237] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e575ee0c8c144974a900494274ac7ce1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44209 } }
I20260620 06:11:43.931707 18276 sys_catalog.cc:564] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:43.932886 18283 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e575ee0c8c144974a900494274ac7ce1. Latest consensus state: current_term: 1 leader_uuid: "e575ee0c8c144974a900494274ac7ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e575ee0c8c144974a900494274ac7ce1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44209 } } }
I20260620 06:11:43.932875 18282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e575ee0c8c144974a900494274ac7ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e575ee0c8c144974a900494274ac7ce1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44209 } } }
I20260620 06:11:43.933069 18283 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:43.933132 18282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:43.933972 18292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:43.936882 18292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:43.941725 18292 catalog_manager.cc:1347] Generated new cluster ID: 4fc9e9402a23421ca3f65085231d40df
I20260620 06:11:43.941798 18292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:43.952329 18292 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:43.953193 18292 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:43.958395 18292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e575ee0c8c144974a900494274ac7ce1: Generated new TSK 0
I20260620 06:11:43.959087 18292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:43.995523 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:44209
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:44.103012 18302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:44.103276 18302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:44.103392 18302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:44.108937 18302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:44.108999 18302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:44.109087 18302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:44.114291 18302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:44.115381 18302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:44.116343 18302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:44.125931 18310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:44.125909 18308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:44.126118 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:44.126192 18302 server_base.cc:1034] running on GCE node
I20260620 06:11:44.126812 18302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:44.127497 18302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:44.128759 18302 hybrid_clock.cc:648] HybridClock initialized: now 1781935904128686 us; error 71 us; skew 500 ppm
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:44.131974 18302 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:44.132930 18302 webserver.cc:460] Webserver started at http://127.14.209.129:43489/ using document root <none> and password file <none>
I20260620 06:11:44.133490 18302 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:44.133584 18302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:44.133805 18302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:44.135917 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "0c85fb85042744138375ebcdc28d868a"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "bf9760578e48536f3ebddf08da88a2df"
server_key_iv: "689a5220dd708837db2505b1833ce355"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.136461 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "0c85fb85042744138375ebcdc28d868a"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "bf9760578e48536f3ebddf08da88a2df"
server_key_iv: "689a5220dd708837db2505b1833ce355"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.140589 18302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 06:11:44.142877 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.144320 18302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:11:44.144465 18302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "0c85fb85042744138375ebcdc28d868a"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "bf9760578e48536f3ebddf08da88a2df"
server_key_iv: "689a5220dd708837db2505b1833ce355"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.144584 18302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:44.158895 18302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:44.162657 18302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:44.162859 18302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:44.163429 18302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:44.164213 18302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:44.164294 18302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.164363 18302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:44.164407 18302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.173280 18302 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:36085
I20260620 06:11:44.173314 18430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:36085 every 8 connection(s)
I20260620 06:11:44.174109 18302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:44.181324 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18302
I20260620 06:11:44.181514 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:44.181804 15174 external_mini_cluster.cc:1428] Setting key 95bd4a7da46279451497f522f0a288f5
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:44.189291 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.176414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:34893 (local address 127.14.209.190:44209)
0620 06:11:44.176574 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:44.176579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:44.177906 (+ 1327us) server_negotiation.cc:407] Connection header received
0620 06:11:44.178832 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:44.178836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:44.178903 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:44.179013 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:44.180174 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.180718 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.181992 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.182277 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.182384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.185407 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:44.185410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:44.185413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:44.185435 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:44.187095 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.187714 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.187717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.187718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.187786 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.188187 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.188189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.188190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.188348 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:44.188423 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:44.189076 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 06:11:44.189175 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:11:44.190392 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.176639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:44209 (local address 127.14.209.129:34893)
0620 06:11:44.177703 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:44.177743 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:44.178618 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:44.179205 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:44.179212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:44.179573 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:44.180019 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:44.180031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.180843 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.180847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:44.181831 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:44.181840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.182448 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.182453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:44.182660 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.183407 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:44.183429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:44.185215 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:44.187223 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:44.187232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:44.187250 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:44.187612 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:44.187882 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:44.187889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:44.187894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:44.188085 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:44.188454 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:44.188463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:44.188796 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 06:11:44.189025 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 06:11:44.189368 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":561,"threads_started":1}
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:44.191721 18431 heartbeater.cc:347] Connected to a master server at 127.14.209.190:44209
I20260620 06:11:44.191931 18431 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:44.192344 18431 heartbeater.cc:511] Master 127.14.209.190:44209 requested a full tablet report, sending...
I20260620 06:11:44.193694 18235 ts_manager.cc:194] Registered new tserver with Master: 0c85fb85042744138375ebcdc28d868a (127.14.209.129:36085)
I20260620 06:11:44.194823 18235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:34893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:44.215456 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:44209
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:44.320890 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:44.321110 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:44.321151 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:44.325281 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:44.325335 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:44.325390 18436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:44.330338 18436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:44.331359 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:44.332320 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:44.341789 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:44.341789 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:44.341845 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:44.341975 18436 server_base.cc:1034] running on GCE node
I20260620 06:11:44.342455 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:44.343145 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:44.344331 18436 hybrid_clock.cc:648] HybridClock initialized: now 1781935904344298 us; error 48 us; skew 500 ppm
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:44.347601 18436 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:44.348482 18436 webserver.cc:460] Webserver started at http://127.14.209.130:41443/ using document root <none> and password file <none>
I20260620 06:11:44.349022 18436 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:44.349107 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:44.349334 18436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:44.351409 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "d87119805b6b4582b0afa234a93ceef4"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "0cdc064b701c6eeb1928a90fc322f583"
server_key_iv: "44d9df781e3ea1c5d4ea303d6a2655bc"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.351943 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "d87119805b6b4582b0afa234a93ceef4"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "0cdc064b701c6eeb1928a90fc322f583"
server_key_iv: "44d9df781e3ea1c5d4ea303d6a2655bc"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.355453 18436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:44.357733 18451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.358884 18436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:44.359020 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "d87119805b6b4582b0afa234a93ceef4"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "0cdc064b701c6eeb1928a90fc322f583"
server_key_iv: "44d9df781e3ea1c5d4ea303d6a2655bc"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.359189 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:44.374313 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:44.377158 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:44.377347 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:44.377914 18436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:44.378720 18436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:44.378789 18436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.378860 18436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:44.378911 18436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.387971 18436 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:45211
I20260620 06:11:44.387993 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:45211 every 8 connection(s)
I20260620 06:11:44.388764 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:44.391480 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18436
I20260620 06:11:44.391620 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:44.391901 15174 external_mini_cluster.cc:1428] Setting key 26f62c615a3644c133028325e908dfa9
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:11:44.403820 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.390899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:47765 (local address 127.14.209.190:44209)
0620 06:11:44.391044 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:44.391072 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:44.391987 (+ 915us) server_negotiation.cc:407] Connection header received
0620 06:11:44.393585 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:44.393589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:44.393637 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:44.393728 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:44.394840 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.395451 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.396905 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.397325 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.397456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.400217 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:44.400220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:44.400222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:44.400244 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:44.401774 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.402212 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.402215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.402216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.402284 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.402639 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.402643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.402645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.402836 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:44.402952 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:44.403537 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 06:11:44.403658 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:11:44.404747 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.391333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:44209 (local address 127.14.209.130:47765)
0620 06:11:44.391814 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:44.391862 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:44.393379 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:44.393860 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:44.393866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:44.394208 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:44.394654 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:44.394671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.395579 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.395585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:44.396688 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:44.396697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.397487 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.397490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:44.397652 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.398244 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:44.398260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:44.400029 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:44.401881 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:44.401885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:44.401895 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:44.402129 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:44.402363 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:44.402365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:44.402366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:44.402484 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:44.402973 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:44.402979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:44.403252 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 06:11:44.403481 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 06:11:44.403802 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260620 06:11:44.405877 18565 heartbeater.cc:347] Connected to a master server at 127.14.209.190:44209
I20260620 06:11:44.406101 18565 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:44.406543 18565 heartbeater.cc:511] Master 127.14.209.190:44209 requested a full tablet report, sending...
I20260620 06:11:44.407565 18235 ts_manager.cc:194] Registered new tserver with Master: d87119805b6b4582b0afa234a93ceef4 (127.14.209.130:45211)
I20260620 06:11:44.408404 18235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:47765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:44.421664 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:44209
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:44.529428 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:44.529652 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:44.529693 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:44.533704 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:44.533753 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:44.533809 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:44.538800 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:44.539695 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:44.540565 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:11:44.549993 18570 server_base.cc:1034] running on GCE node
W20260620 06:11:44.549958 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:44.549994 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:44.550017 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:44.550678 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:44.551474 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:44.552800 18570 hybrid_clock.cc:648] HybridClock initialized: now 1781935904552775 us; error 38 us; skew 500 ppm
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:44.555497 18570 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:44.556262 18570 webserver.cc:460] Webserver started at http://127.14.209.131:35495/ using document root <none> and password file <none>
I20260620 06:11:44.556792 18570 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:44.556882 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:44.557092 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:44.558894 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "59871ba238574925ac6ce9ed321f89cc"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "14532c25460161850039b3949c57aa36"
server_key_iv: "62c26a108ef44da5610cbacfd58d4ebf"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.559468 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "59871ba238574925ac6ce9ed321f89cc"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "14532c25460161850039b3949c57aa36"
server_key_iv: "62c26a108ef44da5610cbacfd58d4ebf"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.562819 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:44.564998 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.566059 18570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:44.566181 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "59871ba238574925ac6ce9ed321f89cc"
format_stamp: "Formatted at 2026-06-20 06:11:44 on dist-test-slave-zt01"
server_key: "14532c25460161850039b3949c57aa36"
server_key_iv: "62c26a108ef44da5610cbacfd58d4ebf"
server_key_version: "encryptionkey@0"
I20260620 06:11:44.566280 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:44.592948 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:44.595667 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:44.595863 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:44.596412 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:44.597290 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:44.597362 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.597438 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:44.597486 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:44.606673 18570 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:41761
I20260620 06:11:44.606722 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:41761 every 8 connection(s)
I20260620 06:11:44.607692 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:44.617574 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18570
I20260620 06:11:44.617695 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:44.617998 15174 external_mini_cluster.cc:1428] Setting key 3e79060f6c2b4baf2a1399beb67d801c
I20260620 06:11:44.620942 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.609522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:33915 (local address 127.14.209.190:44209)
0620 06:11:44.609686 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:44.609692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:44.610412 (+ 720us) server_negotiation.cc:407] Connection header received
0620 06:11:44.611768 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:44.611771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:44.611817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:44.611902 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:44.612991 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.613539 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.614631 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.614861 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.614953 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.617293 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:44.617296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:44.617298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:44.617321 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:44.618925 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.619524 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.619527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.619529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.619619 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.620069 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.620072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.620073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.620236 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:44.620310 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:44.620725 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 06:11:44.620819 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:11:44.621608 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.609802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:44209 (local address 127.14.209.131:33915)
0620 06:11:44.610266 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:44.610305 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:44.611575 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:44.612064 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:44.612070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:44.612398 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:44.612833 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:44.612845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.613657 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.613660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:44.614513 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:44.614522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.615009 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.615011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:44.615202 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.615705 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:44.615719 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:44.617147 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:44.619035 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:44.619041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:44.619095 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:44.619431 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:44.619709 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:44.619711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:44.619712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:44.619850 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:44.620325 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:44.620329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:44.620522 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 06:11:44.620662 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:11:44.620872 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":110,"threads_started":1}
I20260620 06:11:44.622594 18699 heartbeater.cc:347] Connected to a master server at 127.14.209.190:44209
I20260620 06:11:44.622803 18699 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:44.623291 18699 heartbeater.cc:511] Master 127.14.209.190:44209 requested a full tablet report, sending...
I20260620 06:11:44.624238 18235 ts_manager.cc:194] Registered new tserver with Master: 59871ba238574925ac6ce9ed321f89cc (127.14.209.131:41761)
I20260620 06:11:44.624869 18235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:33915
I20260620 06:11:44.631460 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935904, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:44.655710 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:44.647300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.14.209.190:44209)
0620 06:11:44.647441 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:44.647447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:44.647610 (+ 163us) server_negotiation.cc:407] Connection header received
0620 06:11:44.647753 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:44.647756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:44.647794 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:44.647884 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:44.648607 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.649125 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.649963 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:44.650179 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:44.650285 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:44.652602 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:44.652605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:44.652607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:44.652627 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:44.653994 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.654357 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.654359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.654360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.654425 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:44.654662 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:44.654664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:44.654665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:44.654853 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:44.654950 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:44.655455 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 06:11:44.655593 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:11:44.656044 18235 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37440
I20260620 06:11:44.656894 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18302
I20260620 06:11:44.662354 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18436
I20260620 06:11:44.667776 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18570
I20260620 06:11:44.673269 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18208
2026-06-20T06:11:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2097 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18711](info): set up 2 sockets
Jun 20 06:11:44 dist-test-slave-zt01 krb5kdc[18711](info): commencing operation
krb5kdc: starting...
W20260620 06:11:45.711151 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:45 dist-test-slave-zt01 krb5kdc[18711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:45.831959 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34513
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:45.938967 18724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:45.939231 18724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:45.939270 18724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:45.943121 18724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:45.943169 18724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:45.943189 18724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:45.943205 18724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:45.943220 18724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:45.948288 18724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:45.949179 18724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:45.950080 18724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:45.958051 18730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:45.958156 18732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:45.958058 18729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:45.958310 18724 server_base.cc:1034] running on GCE node
I20260620 06:11:45.958917 18724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:45.959971 18724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:45.961170 18724 hybrid_clock.cc:648] HybridClock initialized: now 1781935905961140 us; error 44 us; skew 500 ppm
Jun 20 06:11:45 dist-test-slave-zt01 krb5kdc[18711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:45.964054 18724 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:45.965005 18724 webserver.cc:460] Webserver started at http://127.14.209.190:36539/ using document root <none> and password file <none>
I20260620 06:11:45.966053 18724 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:45.966158 18724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:45.966408 18724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:45.969693 18724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "7c092d4884304119b5b096bb24cac434"
format_stamp: "Formatted at 2026-06-20 06:11:45 on dist-test-slave-zt01"
server_key: "7934731e4d804eaa85def16c56afc4be"
server_key_iv: "7c1ef403c77d83cfbcf217cf2c7c9934"
server_key_version: "encryptionkey@0"
I20260620 06:11:45.970615 18724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "7c092d4884304119b5b096bb24cac434"
format_stamp: "Formatted at 2026-06-20 06:11:45 on dist-test-slave-zt01"
server_key: "7934731e4d804eaa85def16c56afc4be"
server_key_iv: "7c1ef403c77d83cfbcf217cf2c7c9934"
server_key_version: "encryptionkey@0"
I20260620 06:11:45.976281 18724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 06:11:45.980198 18739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:45.981738 18724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.001s
I20260620 06:11:45.981887 18724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "7c092d4884304119b5b096bb24cac434"
format_stamp: "Formatted at 2026-06-20 06:11:45 on dist-test-slave-zt01"
server_key: "7934731e4d804eaa85def16c56afc4be"
server_key_iv: "7c1ef403c77d83cfbcf217cf2c7c9934"
server_key_version: "encryptionkey@0"
I20260620 06:11:45.982019 18724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:46.005010 18724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:46.009150 18724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:46.009402 18724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:46.016997 18724 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:34513
I20260620 06:11:46.017011 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:34513 every 8 connection(s)
I20260620 06:11:46.017850 18724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:46.020390 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:46.025406 18792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434: Bootstrap starting.
I20260620 06:11:46.027663 18792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:46.028003 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18724
I20260620 06:11:46.028122 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:46.028388 15174 external_mini_cluster.cc:1428] Setting key 531e593467aa6480aff4db467c85ee94
I20260620 06:11:46.028494 18792 log.cc:824] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:46.030552 18792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434: No bootstrap required, opened a new log
I20260620 06:11:46.034644 18792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c092d4884304119b5b096bb24cac434" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34513 } }
I20260620 06:11:46.034917 18792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:46.034969 18792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c092d4884304119b5b096bb24cac434, State: Initialized, Role: FOLLOWER
I20260620 06:11:46.035441 18792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c092d4884304119b5b096bb24cac434" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34513 } }
I20260620 06:11:46.035609 18792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:46.035697 18792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:46.035820 18792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:46.037019 18792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c092d4884304119b5b096bb24cac434" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34513 } }
I20260620 06:11:46.037336 18792 leader_election.cc:302] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c092d4884304119b5b096bb24cac434; no voters:
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:46.037640 18792 leader_election.cc:288] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:46.037734 18797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:46.037966 18797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [term 1 LEADER]: Becoming Leader. State: Replica: 7c092d4884304119b5b096bb24cac434, State: Running, Role: LEADER
I20260620 06:11:46.038296 18797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c092d4884304119b5b096bb24cac434" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34513 } }
I20260620 06:11:46.038884 18792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:46.039832 18798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c092d4884304119b5b096bb24cac434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c092d4884304119b5b096bb24cac434" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34513 } } }
I20260620 06:11:46.039979 18798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:46.040354 18799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c092d4884304119b5b096bb24cac434. Latest consensus state: current_term: 1 leader_uuid: "7c092d4884304119b5b096bb24cac434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c092d4884304119b5b096bb24cac434" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34513 } } }
I20260620 06:11:46.040601 18799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:46.043586 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.029435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.14.209.190:34513)
0620 06:11:46.029783 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:46.029800 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:46.029836 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:11:46.030636 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:46.030656 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:46.031019 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:46.031371 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:46.032028 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.033508 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.034319 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.035018 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.035277 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.038139 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:46.038146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:46.038160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:46.038188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:46.040328 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.040870 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.040875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.040880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.041272 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.041637 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.041640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.041642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.042020 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:46.042218 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:46.042518 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 06:11:46.042887 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
W20260620 06:11:46.043886 18813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:46.043960 18813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:46.044018 18806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:46.044812 18806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:46.050236 18806 catalog_manager.cc:1347] Generated new cluster ID: 99fe5f5b52e04972903a0be1d84ffd0e
I20260620 06:11:46.050295 18806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:46.063122 18806 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:46.063966 18806 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:46.074120 18806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c092d4884304119b5b096bb24cac434: Generated new TSK 0
I20260620 06:11:46.074878 18806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:46.110747 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34513
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:46.224638 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:46.224859 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:46.224915 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:46.228850 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:46.228900 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:46.228988 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:46.233919 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:46.234864 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:46.235831 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:46.245261 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:46.245232 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:46.245332 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:46.245532 18818 server_base.cc:1034] running on GCE node
I20260620 06:11:46.245919 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:46.246495 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:46.247660 18818 hybrid_clock.cc:648] HybridClock initialized: now 1781935906247646 us; error 26 us; skew 500 ppm
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:46.250321 18818 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:46.251225 18818 webserver.cc:460] Webserver started at http://127.14.209.129:46375/ using document root <none> and password file <none>
I20260620 06:11:46.251742 18818 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:46.251828 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:46.252033 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:46.253919 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "dc8b0e2f8e2e48d8896edcfeeb4244fa"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "19f14034cab96e8300e7bb7dbceed37a"
server_key_iv: "3988750d8d7cccd8d0595eb106286686"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.254436 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "dc8b0e2f8e2e48d8896edcfeeb4244fa"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "19f14034cab96e8300e7bb7dbceed37a"
server_key_iv: "3988750d8d7cccd8d0595eb106286686"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.258332 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:11:46.260639 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.261778 18818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:46.261950 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "dc8b0e2f8e2e48d8896edcfeeb4244fa"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "19f14034cab96e8300e7bb7dbceed37a"
server_key_iv: "3988750d8d7cccd8d0595eb106286686"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.262060 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:46.277913 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:46.281050 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:46.281328 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:46.281903 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:46.282809 18818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:46.282886 18818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.282948 18818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:46.282998 18818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.293390 18818 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:37111
I20260620 06:11:46.293462 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:37111 every 8 connection(s)
I20260620 06:11:46.294287 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:46.297125 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18818
I20260620 06:11:46.297262 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:46.297571 15174 external_mini_cluster.cc:1428] Setting key 33db6a1ee09344a92acd915796c4f950
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:46.310392 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.296946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:56847 (local address 127.14.209.190:34513)
0620 06:11:46.297107 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:46.297112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:46.298306 (+ 1194us) server_negotiation.cc:407] Connection header received
0620 06:11:46.299774 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:46.299780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:46.299841 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:46.299934 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:46.301342 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.301880 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.303029 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.303480 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.303603 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.306380 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:46.306385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:46.306387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:46.306419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:46.308305 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.308795 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.308798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.308799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.308878 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.309328 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.309331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.309332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.309500 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:46.309599 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:46.310136 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 06:11:46.310272 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:11:46.311218 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.297391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:34513 (local address 127.14.209.129:56847)
0620 06:11:46.298144 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:46.298194 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:46.299576 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:46.300086 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:46.300092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:46.300510 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:46.301184 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:46.301203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.301990 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.301993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:46.302868 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:46.302877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.303621 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.303624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:46.303765 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.304411 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:46.304438 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:46.306046 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:46.308446 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:46.308450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:46.308461 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:46.308698 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:46.308960 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:46.308962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:46.308964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:46.309121 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:46.309639 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:46.309644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:46.309856 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 06:11:46.310009 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:11:46.310277 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":103,"threads_started":1}
I20260620 06:11:46.313160 18947 heartbeater.cc:347] Connected to a master server at 127.14.209.190:34513
I20260620 06:11:46.313413 18947 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:46.313889 18947 heartbeater.cc:511] Master 127.14.209.190:34513 requested a full tablet report, sending...
I20260620 06:11:46.315917 18756 ts_manager.cc:194] Registered new tserver with Master: dc8b0e2f8e2e48d8896edcfeeb4244fa (127.14.209.129:37111)
I20260620 06:11:46.317584 18756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:56847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:46.329648 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34513
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:46.433218 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:46.433441 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:46.433481 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:46.437340 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:46.437386 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:46.437439 18952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:46.442526 18952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:46.443360 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:46.444201 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:46.452958 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:46.452988 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:46.453107 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:46.453275 18952 server_base.cc:1034] running on GCE node
I20260620 06:11:46.453742 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:46.454308 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:46.455497 18952 hybrid_clock.cc:648] HybridClock initialized: now 1781935906455482 us; error 31 us; skew 500 ppm
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:46.458132 18952 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:46.458899 18952 webserver.cc:460] Webserver started at http://127.14.209.130:33419/ using document root <none> and password file <none>
I20260620 06:11:46.459460 18952 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:46.459582 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:46.459792 18952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:46.461676 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "387211ad1d5247a0866cad4d82d195b8"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "c82dcbe18a033e8f1ed43612f8fef3c7"
server_key_iv: "0dd304b88330852947b5887981a90d08"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.462201 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "387211ad1d5247a0866cad4d82d195b8"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "c82dcbe18a033e8f1ed43612f8fef3c7"
server_key_iv: "0dd304b88330852947b5887981a90d08"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.465713 18952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:46.468034 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.469168 18952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:46.469305 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "387211ad1d5247a0866cad4d82d195b8"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "c82dcbe18a033e8f1ed43612f8fef3c7"
server_key_iv: "0dd304b88330852947b5887981a90d08"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.469411 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:46.486541 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:46.489390 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:46.489528 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:46.489984 18952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:46.490674 18952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:46.490718 18952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.490754 18952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:46.490769 18952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.499162 18952 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:41991
I20260620 06:11:46.499212 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:41991 every 8 connection(s)
I20260620 06:11:46.499963 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:46.505151 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 18952
I20260620 06:11:46.505275 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:46.505568 15174 external_mini_cluster.cc:1428] Setting key e207e1cba02914a534fe1c38d2d4d9ed
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:46.513790 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.501583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:45833 (local address 127.14.209.190:34513)
0620 06:11:46.501711 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:46.501715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:46.502451 (+ 736us) server_negotiation.cc:407] Connection header received
0620 06:11:46.503846 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:46.503849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:46.503904 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:46.503986 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:46.505037 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.505716 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.507272 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.507511 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.507619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.510252 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:46.510258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:46.510264 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:46.510296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:46.511833 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.512308 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.512314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.512316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.512423 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.512762 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.512765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.512767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.512948 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:46.513057 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:46.513512 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 06:11:46.513625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:46.514560 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.501889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:34513 (local address 127.14.209.130:45833)
0620 06:11:46.502316 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:46.502354 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:46.503652 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:46.504135 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:46.504141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:46.504473 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:46.504889 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:46.504909 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.505832 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.505836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:46.507154 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:46.507168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.507647 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.507651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:46.507820 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.508418 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:46.508443 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:46.510096 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:46.511937 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:46.511942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:46.511955 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:46.512193 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:46.512519 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:46.512524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:46.512526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:46.512656 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:46.513053 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:46.513059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:46.513278 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 06:11:46.513458 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 06:11:46.513744 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:11:46.515659 19081 heartbeater.cc:347] Connected to a master server at 127.14.209.190:34513
I20260620 06:11:46.515902 19081 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:46.516368 19081 heartbeater.cc:511] Master 127.14.209.190:34513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:11:46.517446 18756 ts_manager.cc:194] Registered new tserver with Master: 387211ad1d5247a0866cad4d82d195b8 (127.14.209.130:41991)
I20260620 06:11:46.518229 18756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:45833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:46.534248 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34513
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:46.637877 19086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:46.638103 19086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:46.638142 19086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:46.642170 19086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:46.642223 19086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:46.642278 19086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:46.647153 19086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:46.648012 19086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:46.648880 19086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:46.657650 19094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:46.657626 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:46.657635 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:46.657838 19086 server_base.cc:1034] running on GCE node
I20260620 06:11:46.658335 19086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:46.659009 19086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:46.660210 19086 hybrid_clock.cc:648] HybridClock initialized: now 1781935906660193 us; error 29 us; skew 500 ppm
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:46.662782 19086 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:46.663640 19086 webserver.cc:460] Webserver started at http://127.14.209.131:46305/ using document root <none> and password file <none>
I20260620 06:11:46.664168 19086 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:46.664255 19086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:46.664471 19086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:46.666354 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "165b8514cb7b4a57a951ae2cbfec29b4"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "21bf7f0a3451a731ea790fbc08f9dacd"
server_key_iv: "39a7fc1192ddba7f87aed843c741ccf4"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.666879 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "165b8514cb7b4a57a951ae2cbfec29b4"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "21bf7f0a3451a731ea790fbc08f9dacd"
server_key_iv: "39a7fc1192ddba7f87aed843c741ccf4"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.670332 19086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:46.672724 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.674006 19086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:46.674141 19086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "165b8514cb7b4a57a951ae2cbfec29b4"
format_stamp: "Formatted at 2026-06-20 06:11:46 on dist-test-slave-zt01"
server_key: "21bf7f0a3451a731ea790fbc08f9dacd"
server_key_iv: "39a7fc1192ddba7f87aed843c741ccf4"
server_key_version: "encryptionkey@0"
I20260620 06:11:46.674247 19086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:46.688915 19086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:46.692037 19086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:46.692221 19086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:46.692739 19086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:46.693535 19086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:46.693599 19086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.693663 19086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:46.693714 19086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:46.702134 19086 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:43305
I20260620 06:11:46.702180 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:43305 every 8 connection(s)
I20260620 06:11:46.702901 19086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:46.709564 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19086
I20260620 06:11:46.709685 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:46.709959 15174 external_mini_cluster.cc:1428] Setting key 0b9555201e7b8d1bc053259622d3f0e7
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:46.715883 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.704467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:58629 (local address 127.14.209.190:34513)
0620 06:11:46.704606 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:46.704610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:46.705353 (+ 743us) server_negotiation.cc:407] Connection header received
0620 06:11:46.706674 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:46.706677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:46.706732 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:46.706842 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:46.707936 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.708467 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.709554 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.709884 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.709992 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.712673 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:46.712677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:46.712680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:46.712704 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:46.713934 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.714415 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.714421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.714422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.714507 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.714841 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.714847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.714850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.715028 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:46.715158 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:46.715611 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 06:11:46.715723 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:11:46.716548 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.704780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:34513 (local address 127.14.209.131:58629)
0620 06:11:46.705207 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:46.705245 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:46.706463 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:46.706996 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:46.707003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:46.707369 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:46.707799 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:46.707811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.708589 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.708595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:46.709423 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:46.709432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.710027 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.710031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:46.710193 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.710776 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:46.710818 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:46.712519 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:46.714055 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:46.714060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:46.714073 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:46.714311 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:46.714596 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:46.714600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:46.714602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:46.714735 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:46.715154 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:46.715160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:46.715381 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:11:46.715557 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 06:11:46.715835 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":121,"threads_started":1}
I20260620 06:11:46.717593 19215 heartbeater.cc:347] Connected to a master server at 127.14.209.190:34513
I20260620 06:11:46.717782 19215 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:46.718161 19215 heartbeater.cc:511] Master 127.14.209.190:34513 requested a full tablet report, sending...
I20260620 06:11:46.719039 18756 ts_manager.cc:194] Registered new tserver with Master: 165b8514cb7b4a57a951ae2cbfec29b4 (127.14.209.131:43305)
I20260620 06:11:46.719586 18756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:58629
I20260620 06:11:46.723819 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:46.749155 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.739558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.14.209.129:37111)
0620 06:11:46.739877 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:46.739890 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:46.739922 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:11:46.739968 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:46.739973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:46.740173 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:46.740288 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:46.741008 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.741752 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.742590 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.742920 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.743012 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.745916 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:46.745924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:46.745934 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:46.745954 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:46.747544 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.747934 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.747938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.747940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.748009 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.748318 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.748320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.748321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.748515 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:46.748668 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:46.748879 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 06:11:46.748989 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":120,"threads_started":1}
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[18711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:46.759626 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:46.751799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.14.209.190:34513)
0620 06:11:46.751986 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:46.751991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:46.752040 (+ 49us) server_negotiation.cc:407] Connection header received
0620 06:11:46.752180 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:46.752183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:46.752222 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:46.752306 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:46.752891 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.753382 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.754176 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:46.754403 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:46.754509 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:46.756756 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:46.756759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:46.756761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:46.756780 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:46.758167 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.758532 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.758534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.758535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.758601 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:46.758815 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:46.758817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:46.758818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:46.758997 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:46.759161 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:46.759329 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 06:11:46.759412 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 06:11:46.759965 18756 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51658
I20260620 06:11:46.760475 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18818
I20260620 06:11:46.765691 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18952
I20260620 06:11:46.770555 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19086
I20260620 06:11:46.775597 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 18724
2026-06-20T06:11:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2102 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[19227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[19227](info): set up 2 sockets
Jun 20 06:11:46 dist-test-slave-zt01 krb5kdc[19227](info): commencing operation
krb5kdc: starting...
W20260620 06:11:47.827488 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:47 dist-test-slave-zt01 krb5kdc[19227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:47.949683 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:37697
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:37697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:48.053195 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:48.053442 19240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:48.053507 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:48.057487 19240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:48.057569 19240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:48.057606 19240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:48.057642 19240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:48.057680 19240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:48.062672 19240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:37697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:48.063625 19240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:48.064604 19240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:48.072455 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.072435 19240 server_base.cc:1034] running on GCE node
W20260620 06:11:48.072472 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:48.072403 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.073208 19240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:48.074168 19240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:48.075328 19240 hybrid_clock.cc:648] HybridClock initialized: now 1781935908075313 us; error 34 us; skew 500 ppm
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:48.081182 19240 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:48.081853 19240 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:48.081947 19240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:48.082168 19240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:48.084185 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "e90d2742b7c74c67a9152fdd3d30c6fc"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "cc6c1c7ed0aee2501d5fc0a33f370a05"
server_key_iv: "88261b6b93770255a88aa5c42858a4e7"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.084764 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "e90d2742b7c74c67a9152fdd3d30c6fc"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "cc6c1c7ed0aee2501d5fc0a33f370a05"
server_key_iv: "88261b6b93770255a88aa5c42858a4e7"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.088580 19240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:11:48.091172 19254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.092367 19240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:48.092538 19240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "e90d2742b7c74c67a9152fdd3d30c6fc"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "cc6c1c7ed0aee2501d5fc0a33f370a05"
server_key_iv: "88261b6b93770255a88aa5c42858a4e7"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.092664 19240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:48.116560 19240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:48.120033 19240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:48.120236 19240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:48.127254 19240 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:37697
I20260620 06:11:48.127533 19306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:37697 every 8 connection(s)
I20260620 06:11:48.127916 19240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:48.130514 19307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:48.135154 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19240
I20260620 06:11:48.135337 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:48.135608 15174 external_mini_cluster.cc:1428] Setting key e6463654fa84c87a3775ea89151d202f
I20260620 06:11:48.135810 19307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: Bootstrap starting.
I20260620 06:11:48.138346 19307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:48.139202 19307 log.cc:824] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:48.140997 19307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: No bootstrap required, opened a new log
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:48.145538 19307 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37697 } }
I20260620 06:11:48.145826 19307 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:48.145884 19307 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90d2742b7c74c67a9152fdd3d30c6fc, State: Initialized, Role: FOLLOWER
I20260620 06:11:48.146363 19307 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37697 } }
I20260620 06:11:48.146538 19307 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:48.146607 19307 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:48.146700 19307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:48.147617 19307 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37697 } }
I20260620 06:11:48.147951 19307 leader_election.cc:302] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90d2742b7c74c67a9152fdd3d30c6fc; no voters:
I20260620 06:11:48.148286 19307 leader_election.cc:288] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:48.148350 19312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:48.148591 19312 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [term 1 LEADER]: Becoming Leader. State: Replica: e90d2742b7c74c67a9152fdd3d30c6fc, State: Running, Role: LEADER
I20260620 06:11:48.148927 19312 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37697 } }
I20260620 06:11:48.149555 19307 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:48.151185 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.137035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.14.209.190:37697)
0620 06:11:48.137459 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.137476 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.137516 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:11:48.138336 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.138360 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.138763 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.139133 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:48.139874 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.140802 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.141936 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.142408 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.142532 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.145768 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:48.145773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:48.145785 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:48.145810 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:48.147900 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.148377 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.148382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.148388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.148555 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.148876 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.148879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.148881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.149314 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:48.149452 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.149925 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.150267 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260620 06:11:48.151615 19314 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37697 } } }
I20260620 06:11:48.151695 19313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90d2742b7c74c67a9152fdd3d30c6fc. Latest consensus state: current_term: 1 leader_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d2742b7c74c67a9152fdd3d30c6fc" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37697 } } }
I20260620 06:11:48.151997 19313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:48.152242 19314 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:48.154546 19326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:48.154626 19326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:48.154680 19327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:48.155494 19327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:48.161227 19327 catalog_manager.cc:1347] Generated new cluster ID: 7f7b6ec4b4864aefadd2ccd428da549e
I20260620 06:11:48.161298 19327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:48.174984 19327 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:48.175947 19327 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:48.188294 19327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: Generated new TSK 0
I20260620 06:11:48.188917 19327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:48.226934 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:37697
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:48.333148 19333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:48.333379 19333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:48.333420 19333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:48.337493 19333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:48.337546 19333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:48.337638 19333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:48.342628 19333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:48.343633 19333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:48.344571 19333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:48.353787 19339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:48.353785 19338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.353897 19333 server_base.cc:1034] running on GCE node
W20260620 06:11:48.353787 19341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.354421 19333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:48.355001 19333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:48.356217 19333 hybrid_clock.cc:648] HybridClock initialized: now 1781935908356203 us; error 25 us; skew 500 ppm
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:48.358966 19333 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:48.359653 19333 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:48.359714 19333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:48.359864 19333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:48.361670 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "c63761a519dd4d6b8acaff3124bf9e28"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "445663bbb513e14b25e8b58a571f8e6f"
server_key_iv: "aa7f7e8005a185c5ca22ebd2332ba4b9"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.362208 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "c63761a519dd4d6b8acaff3124bf9e28"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "445663bbb513e14b25e8b58a571f8e6f"
server_key_iv: "aa7f7e8005a185c5ca22ebd2332ba4b9"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.365621 19333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 06:11:48.367859 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.368913 19333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:11:48.369004 19333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "c63761a519dd4d6b8acaff3124bf9e28"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "445663bbb513e14b25e8b58a571f8e6f"
server_key_iv: "aa7f7e8005a185c5ca22ebd2332ba4b9"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.369133 19333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:48.384254 19333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:48.387444 19333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:48.387625 19333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:48.388167 19333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:48.388934 19333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:48.388980 19333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.389051 19333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:48.389084 19333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.397732 19333 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:35429
I20260620 06:11:48.397753 19460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:35429 every 8 connection(s)
I20260620 06:11:48.398281 19333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:48.402683 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19333
I20260620 06:11:48.402812 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:48.403112 15174 external_mini_cluster.cc:1428] Setting key 6e7c49919f39cb610fc29fa07d35a445
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:48.411355 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.399904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:54913 (local address 127.14.209.190:37697)
0620 06:11:48.400031 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.400035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.401072 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 06:11:48.401875 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.401878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.401924 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.401999 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:48.403443 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.404026 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.405156 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.405410 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.405529 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.407913 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:48.407917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:48.407921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:48.407949 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:48.409379 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.409872 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.409877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.409879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.409957 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.410303 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.410308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.410309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.410476 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:48.410585 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.411071 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.411190 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:11:48.412108 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.400154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:37697 (local address 127.14.209.129:54913)
0620 06:11:48.400937 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.400974 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.401675 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.402146 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.402152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.402602 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:48.403289 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.403309 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.404154 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.404158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.405026 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.405035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.405553 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.405557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.405732 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.406290 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:48.406305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:48.407758 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:48.409501 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:48.409507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:48.409528 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:48.409770 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:48.410068 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:48.410071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:48.410073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:48.410208 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:48.410584 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:48.410590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:48.410814 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.411000 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.411313 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":105,"threads_started":1}
I20260620 06:11:48.413301 19461 heartbeater.cc:347] Connected to a master server at 127.14.209.190:37697
I20260620 06:11:48.413550 19461 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:11:48.414055 19461 heartbeater.cc:511] Master 127.14.209.190:37697 requested a full tablet report, sending...
I20260620 06:11:48.415593 19271 ts_manager.cc:194] Registered new tserver with Master: c63761a519dd4d6b8acaff3124bf9e28 (127.14.209.129:35429)
I20260620 06:11:48.416667 19271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:54913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:48.430372 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:37697
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:48.536976 19466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:48.537221 19466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:48.537263 19466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:48.541172 19466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:48.541218 19466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:48.541275 19466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:48.546260 19466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:48.547122 19466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:48.547987 19466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:48.557111 19474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:48.557070 19471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:48.557073 19472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.557277 19466 server_base.cc:1034] running on GCE node
I20260620 06:11:48.557714 19466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:48.558322 19466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:48.559473 19466 hybrid_clock.cc:648] HybridClock initialized: now 1781935908559454 us; error 29 us; skew 500 ppm
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:48.562086 19466 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:48.562693 19466 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:48.562752 19466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:48.562898 19466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:48.564779 19466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "61e041649bd140579b4a8701722419ee"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "66d91445aab19bdc7fd9fbbf4575e1ab"
server_key_iv: "aad270bfe1f6f9bafe72a0c45b8e21cd"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.565250 19466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "61e041649bd140579b4a8701722419ee"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "66d91445aab19bdc7fd9fbbf4575e1ab"
server_key_iv: "aad270bfe1f6f9bafe72a0c45b8e21cd"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.568611 19466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:48.570767 19480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.571910 19466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:48.572000 19466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "61e041649bd140579b4a8701722419ee"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "66d91445aab19bdc7fd9fbbf4575e1ab"
server_key_iv: "aad270bfe1f6f9bafe72a0c45b8e21cd"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.572124 19466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:48.591684 19466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:48.594623 19466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:48.594810 19466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:48.595415 19466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:48.596261 19466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:48.596352 19466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.596422 19466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:48.596465 19466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.605176 19466 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:38213
I20260620 06:11:48.605239 19593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:38213 every 8 connection(s)
I20260620 06:11:48.605716 19466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:48.605914 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19466
I20260620 06:11:48.606067 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:48.606369 15174 external_mini_cluster.cc:1428] Setting key 4cf33e6f809bb1f655f3d1956f5fcb81
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:11:48.620886 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.607451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:43271 (local address 127.14.209.190:37697)
0620 06:11:48.607584 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.607589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.609033 (+ 1444us) server_negotiation.cc:407] Connection header received
0620 06:11:48.610167 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.610173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.610237 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.610342 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:48.611470 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.612041 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.613845 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.614063 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.614144 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.616975 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:48.616984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:48.616986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:48.617009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:48.618666 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.619313 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.619319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.619321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.619410 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.619781 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.619784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.619786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.619976 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:48.620094 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.620686 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.620776 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:48.621985 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.607923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:37697 (local address 127.14.209.130:43271)
0620 06:11:48.608868 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.608921 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.609942 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.610484 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.610490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.610794 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:48.611317 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.611331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.612184 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.612191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.613715 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.613730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.614234 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.614241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.614478 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.615039 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:48.615087 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:48.616788 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:48.618814 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:48.618820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:48.618834 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:48.619203 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:48.619520 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:48.619523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:48.619526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:48.619669 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:48.620092 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:48.620100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:48.620378 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.620612 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.620963 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
I20260620 06:11:48.623284 19594 heartbeater.cc:347] Connected to a master server at 127.14.209.190:37697
I20260620 06:11:48.623544 19594 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:48.624006 19594 heartbeater.cc:511] Master 127.14.209.190:37697 requested a full tablet report, sending...
I20260620 06:11:48.625079 19271 ts_manager.cc:194] Registered new tserver with Master: 61e041649bd140579b4a8701722419ee (127.14.209.130:38213)
I20260620 06:11:48.625890 19271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:43271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:48.634476 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:37697
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:48.745784 19599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:48.746004 19599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:48.746047 19599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:48.750059 19599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:48.750109 19599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:48.750195 19599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:48.755465 19599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:48.756309 19599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:48.757174 19599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:48.765954 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:48.766034 19607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.766139 19599 server_base.cc:1034] running on GCE node
W20260620 06:11:48.765982 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:48.766564 19599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:48.767158 19599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:48.768324 19599 hybrid_clock.cc:648] HybridClock initialized: now 1781935908768306 us; error 29 us; skew 500 ppm
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:48.771199 19599 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:48.771857 19599 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:48.771947 19599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:48.772149 19599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:48.774190 19599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "b45e8b7add924df2a0b4d44634e5bc64"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "e2790735f473d6da902fe5b315d2f8c5"
server_key_iv: "18b0214e0ef58df0f80e63a3a73220d1"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.774726 19599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "b45e8b7add924df2a0b4d44634e5bc64"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "e2790735f473d6da902fe5b315d2f8c5"
server_key_iv: "18b0214e0ef58df0f80e63a3a73220d1"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.778234 19599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:48.780553 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.781678 19599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:48.781807 19599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "b45e8b7add924df2a0b4d44634e5bc64"
format_stamp: "Formatted at 2026-06-20 06:11:48 on dist-test-slave-zt01"
server_key: "e2790735f473d6da902fe5b315d2f8c5"
server_key_iv: "18b0214e0ef58df0f80e63a3a73220d1"
server_key_version: "encryptionkey@0"
I20260620 06:11:48.781910 19599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:48.807725 19599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:48.811338 19599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:48.811530 19599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:48.812095 19599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:48.812896 19599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:48.812980 19599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.813053 19599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:48.813107 19599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:48.821907 19599 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:39681
I20260620 06:11:48.821939 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:39681 every 8 connection(s)
I20260620 06:11:48.822450 19599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:48.830929 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19599
I20260620 06:11:48.831065 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
Jun 20 06:11:48 dist-test-slave-zt01 krb5kdc[19227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935908, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:48.831389 15174 external_mini_cluster.cc:1428] Setting key c8532d1fde59fcf0ba05cf993ff8d2ef
I20260620 06:11:48.835650 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.824177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:38693 (local address 127.14.209.190:37697)
0620 06:11:48.824388 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.824393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.825494 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 06:11:48.826294 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.826297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.826339 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.826412 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:48.827530 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.828041 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.829134 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.829353 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.829443 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.831924 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:48.831935 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:48.831937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:48.831965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:48.833644 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.834133 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.834137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.834139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.834223 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.834562 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.834565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.834567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.834740 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:48.834856 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.835384 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.835493 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 06:11:48.836330 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.824577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:37697 (local address 127.14.209.131:38693)
0620 06:11:48.825353 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.825390 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.826124 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.826568 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.826574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.826896 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:48.827404 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.827417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.828149 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.828151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.828982 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.828991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.829514 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.829517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.829656 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.830194 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:48.830208 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:48.831753 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:48.833757 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:48.833763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:48.833775 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:48.834032 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:48.834312 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:48.834316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:48.834318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:48.834459 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:48.834851 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:48.834857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:48.835129 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.835323 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.835609 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260620 06:11:48.837333 19727 heartbeater.cc:347] Connected to a master server at 127.14.209.190:37697
I20260620 06:11:48.837535 19727 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:48.837917 19727 heartbeater.cc:511] Master 127.14.209.190:37697 requested a full tablet report, sending...
I20260620 06:11:48.839691 19271 ts_manager.cc:194] Registered new tserver with Master: b45e8b7add924df2a0b4d44634e5bc64 (127.14.209.131:39681)
I20260620 06:11:48.840400 19271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:38693
I20260620 06:11:48.844878 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:11:48.854223 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.846776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.14.209.190:37697)
0620 06:11:48.846906 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.846909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.847159 (+ 250us) server_negotiation.cc:407] Connection header received
0620 06:11:48.847256 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.847258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.847292 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.847358 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:48.847973 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.848463 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.849321 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.849537 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.849614 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.850902 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:48.850905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:48.850907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:48.850928 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:48.852442 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.852943 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.852948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.852949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.853110 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:48.853508 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:48.853510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:48.853512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:48.853716 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:48.853817 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.854018 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.854104 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:11:48.856881 19271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:48.859876 19271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:48.874320 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.867892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:39681 (local address 127.0.0.1:60728)
0620 06:11:48.868219 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.868263 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.868384 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.869497 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.869503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.869528 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:48.869664 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.869671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.871152 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.871155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.872522 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.872533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.873717 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.873720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.873875 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.873916 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.874031 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.874151 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":28,"thread_start_us":79,"threads_started":1}
I20260620 06:11:48.874366 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.868734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.14.209.131:39681)
0620 06:11:48.869065 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.869074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.869092 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:11:48.869151 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.869154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.869337 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.869492 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:48.869898 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.871006 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.872653 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.873584 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.873730 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.873864 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.874077 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.874186 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":155,"threads_started":1}
I20260620 06:11:48.874744 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.868028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.14.209.130:38213)
0620 06:11:48.869343 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.869364 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.869386 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:11:48.869481 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.869486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.869672 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.869829 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:48.870066 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.871153 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.872813 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.873615 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.873730 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.873919 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.874393 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.874564 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":87,"threads_started":1}
I20260620 06:11:48.874861 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.868464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:35429 (local address 127.0.0.1:52878)
0620 06:11:48.870091 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.870108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.870212 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.870635 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.870639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.870658 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:48.870754 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.870761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.872210 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.872213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.873310 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.873319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.874446 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.874451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.874562 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.874582 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.874652 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.874714 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1560,"mutex_wait_us":41,"thread_start_us":84,"threads_started":1}
I20260620 06:11:48.875420 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.868612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.14.209.129:35429)
0620 06:11:48.868915 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.868923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.870213 (+ 1290us) server_negotiation.cc:407] Connection header received
0620 06:11:48.870320 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.870325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.870480 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.870627 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:48.870873 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.872049 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.873425 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.874159 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.874268 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.874377 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.874710 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.874819 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260620 06:11:48.875872 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.867948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:38213 (local address 127.0.0.1:47062)
0620 06:11:48.868701 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.868715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.868801 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.869819 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.869823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.869840 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:48.869938 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.869944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.871297 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.871300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.872701 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.872713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.873987 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.873990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.874111 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.874136 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.874212 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.874286 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":25,"thread_start_us":115,"threads_started":1}
I20260620 06:11:48.878410 19661 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed2d0c0be694c2ca319f4a5d6576e53 (DEFAULT_TABLE table=test-table [id=c719de9307614139a725ded02f88f604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:48.878410 19395 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed2d0c0be694c2ca319f4a5d6576e53 (DEFAULT_TABLE table=test-table [id=c719de9307614139a725ded02f88f604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:48.879545 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed2d0c0be694c2ca319f4a5d6576e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:48.879542 19395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed2d0c0be694c2ca319f4a5d6576e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:48.881279 19528 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed2d0c0be694c2ca319f4a5d6576e53 (DEFAULT_TABLE table=test-table [id=c719de9307614139a725ded02f88f604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:48.882076 19528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ed2d0c0be694c2ca319f4a5d6576e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:48.886641 19745 tablet_bootstrap.cc:492] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Bootstrap starting.
I20260620 06:11:48.887034 19744 tablet_bootstrap.cc:492] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: Bootstrap starting.
I20260620 06:11:48.888286 19746 tablet_bootstrap.cc:492] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Bootstrap starting.
I20260620 06:11:48.889984 19746 tablet_bootstrap.cc:654] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:48.890153 19744 tablet_bootstrap.cc:654] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:48.890990 19745 tablet_bootstrap.cc:654] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:48.891152 19746 log.cc:824] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:48.891355 19744 log.cc:824] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:48.892097 19745 log.cc:824] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:48.893150 19744 tablet_bootstrap.cc:492] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: No bootstrap required, opened a new log
I20260620 06:11:48.893150 19746 tablet_bootstrap.cc:492] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: No bootstrap required, opened a new log
I20260620 06:11:48.893350 19744 ts_tablet_manager.cc:1397] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 06:11:48.893345 19746 ts_tablet_manager.cc:1397] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:11:48.893849 19745 tablet_bootstrap.cc:492] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: No bootstrap required, opened a new log
I20260620 06:11:48.894013 19745 ts_tablet_manager.cc:1397] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 06:11:48.896396 19744 raft_consensus.cc:348] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.896664 19744 raft_consensus.cc:374] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:48.896775 19744 raft_consensus.cc:729] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63761a519dd4d6b8acaff3124bf9e28, State: Initialized, Role: FOLLOWER
I20260620 06:11:48.897176 19746 raft_consensus.cc:348] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.897289 19744 consensus_queue.cc:260] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.897444 19746 raft_consensus.cc:374] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:48.897514 19746 raft_consensus.cc:729] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e041649bd140579b4a8701722419ee, State: Initialized, Role: FOLLOWER
I20260620 06:11:48.897627 19745 raft_consensus.cc:348] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.897859 19745 raft_consensus.cc:374] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:48.897945 19745 raft_consensus.cc:729] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45e8b7add924df2a0b4d44634e5bc64, State: Initialized, Role: FOLLOWER
I20260620 06:11:48.897933 19746 consensus_queue.cc:260] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.898384 19744 ts_tablet_manager.cc:1428] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 06:11:48.898587 19594 heartbeater.cc:503] Master 127.14.209.190:37697 was elected leader, sending a full tablet report...
I20260620 06:11:48.898730 19461 heartbeater.cc:503] Master 127.14.209.190:37697 was elected leader, sending a full tablet report...
I20260620 06:11:48.898828 19746 ts_tablet_manager.cc:1428] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:11:48.898348 19745 consensus_queue.cc:260] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
W20260620 06:11:48.899174 19462 tablet.cc:2369] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:48.899930 19727 heartbeater.cc:503] Master 127.14.209.190:37697 was elected leader, sending a full tablet report...
I20260620 06:11:48.899971 19745 ts_tablet_manager.cc:1428] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:11:48.943383 19751 raft_consensus.cc:482] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:48.943604 19751 raft_consensus.cc:504] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.944931 19751 leader_election.cc:288] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b45e8b7add924df2a0b4d44634e5bc64 (127.14.209.131:39681), c63761a519dd4d6b8acaff3124bf9e28 (127.14.209.129:35429)
I20260620 06:11:48.948865 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.945261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:57317 (local address 127.14.209.129:35429)
0620 06:11:48.945377 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.945382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.945694 (+ 312us) server_negotiation.cc:407] Connection header received
0620 06:11:48.945787 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.945790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.945834 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.945920 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:48.946260 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.946863 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.947888 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.948317 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.948415 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.948494 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.948657 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.948745 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:11:48.948869 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.945164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:35429 (local address 127.14.209.130:57317)
0620 06:11:48.945574 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.945587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.945685 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.945978 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.945981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.945996 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:48.946116 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.946122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.946976 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.946978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.947768 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.947777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.948438 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.948445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.948535 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.948559 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.948630 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.948691 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":3,"thread_start_us":142,"threads_started":1}
I20260620 06:11:48.948896 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.945139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:39681 (local address 127.14.209.130:35009)
0620 06:11:48.945405 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:48.945422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:48.945519 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:48.945786 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:48.945790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:48.945809 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:48.945948 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.945955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.946839 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.946842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.947799 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:48.947808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.948495 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.948508 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:48.948629 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.948653 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:11:48.948728 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:11:48.948788 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":44}
I20260620 06:11:48.948943 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.945238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:35009 (local address 127.14.209.131:39681)
0620 06:11:48.945378 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.945383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.945478 (+ 95us) server_negotiation.cc:407] Connection header received
0620 06:11:48.945603 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.945609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.945668 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.945746 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:48.946041 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.946728 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.947950 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.948355 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.948489 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.948550 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.948755 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.948820 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:48.949590 19681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed2d0c0be694c2ca319f4a5d6576e53" candidate_uuid: "61e041649bd140579b4a8701722419ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e8b7add924df2a0b4d44634e5bc64" is_pre_election: true
I20260620 06:11:48.949589 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed2d0c0be694c2ca319f4a5d6576e53" candidate_uuid: "61e041649bd140579b4a8701722419ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63761a519dd4d6b8acaff3124bf9e28" is_pre_election: true
I20260620 06:11:48.949903 19415 raft_consensus.cc:2454] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e041649bd140579b4a8701722419ee in term 0.
I20260620 06:11:48.949903 19681 raft_consensus.cc:2454] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e041649bd140579b4a8701722419ee in term 0.
I20260620 06:11:48.950402 19482 leader_election.cc:302] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e041649bd140579b4a8701722419ee, c63761a519dd4d6b8acaff3124bf9e28; no voters:
I20260620 06:11:48.950667 19751 raft_consensus.cc:2781] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:48.950763 19751 raft_consensus.cc:482] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:48.950851 19751 raft_consensus.cc:3037] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:48.951988 19751 raft_consensus.cc:504] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.952328 19751 leader_election.cc:288] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [CANDIDATE]: Term 1 election: Requested vote from peers b45e8b7add924df2a0b4d44634e5bc64 (127.14.209.131:39681), c63761a519dd4d6b8acaff3124bf9e28 (127.14.209.129:35429)
I20260620 06:11:48.952800 19681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed2d0c0be694c2ca319f4a5d6576e53" candidate_uuid: "61e041649bd140579b4a8701722419ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45e8b7add924df2a0b4d44634e5bc64"
I20260620 06:11:48.952822 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ed2d0c0be694c2ca319f4a5d6576e53" candidate_uuid: "61e041649bd140579b4a8701722419ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63761a519dd4d6b8acaff3124bf9e28"
I20260620 06:11:48.952939 19681 raft_consensus.cc:3037] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:48.952929 19415 raft_consensus.cc:3037] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:48.954166 19415 raft_consensus.cc:2454] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e041649bd140579b4a8701722419ee in term 1.
I20260620 06:11:48.954166 19681 raft_consensus.cc:2454] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e041649bd140579b4a8701722419ee in term 1.
I20260620 06:11:48.954505 19482 leader_election.cc:302] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e041649bd140579b4a8701722419ee, c63761a519dd4d6b8acaff3124bf9e28; no voters:
I20260620 06:11:48.954730 19751 raft_consensus.cc:2781] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:48.954963 19751 raft_consensus.cc:686] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 1 LEADER]: Becoming Leader. State: Replica: 61e041649bd140579b4a8701722419ee, State: Running, Role: LEADER
I20260620 06:11:48.955293 19751 consensus_queue.cc:237] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } }
I20260620 06:11:48.958165 19271 catalog_manager.cc:5495] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 61e041649bd140579b4a8701722419ee (127.14.209.130). New cstate: current_term: 1 leader_uuid: "61e041649bd140579b4a8701722419ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e041649bd140579b4a8701722419ee" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 } health_report { overall_health: UNKNOWN } } }
I20260620 06:11:48.972287 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:48.968849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.14.209.130:38213)
0620 06:11:48.969003 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:48.969010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:48.969028 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:11:48.969137 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:48.969139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:48.969188 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:48.969295 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:48.969528 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.970016 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.970848 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:48.971126 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:48.971227 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:48.971526 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:48.971899 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:48.972023 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:48.972077 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 06:11:48.972149 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:11:48.978797 19415 raft_consensus.cc:1261] T 5ed2d0c0be694c2ca319f4a5d6576e53 P c63761a519dd4d6b8acaff3124bf9e28 [term 1 FOLLOWER]: Refusing update from remote peer 61e041649bd140579b4a8701722419ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:48.978797 19681 raft_consensus.cc:1261] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 1 FOLLOWER]: Refusing update from remote peer 61e041649bd140579b4a8701722419ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:48.979566 19754 consensus_queue.cc:1035] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63761a519dd4d6b8acaff3124bf9e28" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:48.979758 19751 consensus_queue.cc:1035] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45e8b7add924df2a0b4d44634e5bc64" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:48.995581 19270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:48.995751 19270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:48.998095 19270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e90d2742b7c74c67a9152fdd3d30c6fc: Sending DeleteTablet for 3 replicas of tablet 5ed2d0c0be694c2ca319f4a5d6576e53
I20260620 06:11:48.998780 19528 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed2d0c0be694c2ca319f4a5d6576e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:48 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:47062
I20260620 06:11:48.998967 19395 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed2d0c0be694c2ca319f4a5d6576e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:48 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:52878
I20260620 06:11:48.999281 19767 tablet_replica.cc:330] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: stopping tablet replica
I20260620 06:11:48.999540 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19333
I20260620 06:11:48.999600 19767 raft_consensus.cc:2229] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:48.999907 19767 raft_consensus.cc:2258] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:49.000214 19661 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ed2d0c0be694c2ca319f4a5d6576e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:48 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:60728
I20260620 06:11:49.002027 19767 ts_tablet_manager.cc:1905] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:49.005038 19767 ts_tablet_manager.cc:1918] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:49.005146 19767 log.cc:1196] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/5ed2d0c0be694c2ca319f4a5d6576e53
I20260620 06:11:49.005424 19769 tablet_replica.cc:330] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: stopping tablet replica
I20260620 06:11:49.005517 19767 ts_tablet_manager.cc:1939] T 5ed2d0c0be694c2ca319f4a5d6576e53 P 61e041649bd140579b4a8701722419ee: Deleting consensus metadata
I20260620 06:11:49.005867 19769 raft_consensus.cc:2229] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:49.006247 19769 raft_consensus.cc:2258] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:49.006736 19258 catalog_manager.cc:4841] TS 61e041649bd140579b4a8701722419ee (127.14.209.130:38213): tablet 5ed2d0c0be694c2ca319f4a5d6576e53 (table test-table [id=c719de9307614139a725ded02f88f604]) successfully deleted
W20260620 06:11:49.006814 19256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:35429 (error 108)
I20260620 06:11:49.007552 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19466
I20260620 06:11:49.008435 19769 ts_tablet_manager.cc:1905] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:49.010672 19769 ts_tablet_manager.cc:1918] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:49.010740 19769 log.cc:1196] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/5ed2d0c0be694c2ca319f4a5d6576e53
I20260620 06:11:49.010987 19769 ts_tablet_manager.cc:1939] T 5ed2d0c0be694c2ca319f4a5d6576e53 P b45e8b7add924df2a0b4d44634e5bc64: Deleting consensus metadata
I20260620 06:11:49.012302 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.011946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:35429 (local address 127.0.0.1:52894)
0620 06:11:49.012118 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.012201 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:35429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 06:11:49.012401 19255 catalog_manager.cc:4841] TS b45e8b7add924df2a0b4d44634e5bc64 (127.14.209.131:39681): tablet 5ed2d0c0be694c2ca319f4a5d6576e53 (table test-table [id=c719de9307614139a725ded02f88f604]) successfully deleted
W20260620 06:11:49.012632 19256 catalog_manager.cc:4584] TS c63761a519dd4d6b8acaff3124bf9e28 (127.14.209.129:35429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ed2d0c0be694c2ca319f4a5d6576e53: Network error: Client connection negotiation failed: client connection to 127.14.209.129:35429: connect: Connection refused (error 111)
I20260620 06:11:49.013600 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19599
I20260620 06:11:49.018474 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19240
2026-06-20T06:11:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2243 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T06:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:49Z Disabled control of system clock
I20260620 06:11:49.048143 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:43643
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:43643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:11:49.157183 19774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:49.157429 19774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:49.157466 19774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:49.161425 19774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:49.161478 19774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:49.161494 19774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:49.161510 19774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:49.161525 19774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:49.166106 19774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:43643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:43643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:49.167176 19774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:49.168493 19774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:49.176831 19782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.176846 19779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.177016 19780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:49.177322 19774 server_base.cc:1034] running on GCE node
I20260620 06:11:49.177802 19774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:49.178903 19774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:49.180104 19774 hybrid_clock.cc:648] HybridClock initialized: now 1781935909180077 us; error 42 us; skew 500 ppm
I20260620 06:11:49.182062 19774 webserver.cc:460] Webserver started at http://127.14.209.190:43555/ using document root <none> and password file <none>
I20260620 06:11:49.182624 19774 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:49.182721 19774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:49.182953 19774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:49.184940 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "35a5eba96a354e10b8cb04ee86bdf731"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "dc30076c6489fa8649042fa546a8c509"
server_key_iv: "66cb62234877e62e158a06f506bc1bf0"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.185493 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "35a5eba96a354e10b8cb04ee86bdf731"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "dc30076c6489fa8649042fa546a8c509"
server_key_iv: "66cb62234877e62e158a06f506bc1bf0"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.189138 19774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:49.191519 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.192709 19774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:49.192842 19774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "35a5eba96a354e10b8cb04ee86bdf731"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "dc30076c6489fa8649042fa546a8c509"
server_key_iv: "66cb62234877e62e158a06f506bc1bf0"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.192946 19774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:49.197162 19774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:49.197715 19774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:49.197891 19774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:49.204241 19774 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:43643
I20260620 06:11:49.204262 19840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:43643 every 8 connection(s)
I20260620 06:11:49.205219 19774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:49.208261 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:49.213289 19841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731: Bootstrap starting.
I20260620 06:11:49.213354 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19774
I20260620 06:11:49.213476 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:49.213744 15174 external_mini_cluster.cc:1428] Setting key f61a2d464ea3d0ac632e058f6c82ef23
I20260620 06:11:49.215796 19841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:49.216504 19841 log.cc:824] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:49.218281 19841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731: No bootstrap required, opened a new log
I20260620 06:11:49.219393 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.215007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.14.209.190:43643)
0620 06:11:49.215681 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.215697 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.215730 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:49.216419 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.216435 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.216794 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.217356 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.218150 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.218155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.218165 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.218175 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.218229 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.218244 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.218368 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.218452 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.218691 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":127,"threads_started":1}
I20260620 06:11:49.222043 19841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a5eba96a354e10b8cb04ee86bdf731" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43643 } }
I20260620 06:11:49.222258 19841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:49.222334 19841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a5eba96a354e10b8cb04ee86bdf731, State: Initialized, Role: FOLLOWER
I20260620 06:11:49.222720 19841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a5eba96a354e10b8cb04ee86bdf731" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43643 } }
I20260620 06:11:49.222862 19841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:49.222929 19841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:49.223049 19841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:49.223848 19841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a5eba96a354e10b8cb04ee86bdf731" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43643 } }
I20260620 06:11:49.224161 19841 leader_election.cc:302] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a5eba96a354e10b8cb04ee86bdf731; no voters:
I20260620 06:11:49.224438 19841 leader_election.cc:288] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:49.224503 19846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:49.224799 19846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [term 1 LEADER]: Becoming Leader. State: Replica: 35a5eba96a354e10b8cb04ee86bdf731, State: Running, Role: LEADER
I20260620 06:11:49.225081 19846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a5eba96a354e10b8cb04ee86bdf731" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43643 } }
I20260620 06:11:49.225541 19841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:49.226675 19848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a5eba96a354e10b8cb04ee86bdf731. Latest consensus state: current_term: 1 leader_uuid: "35a5eba96a354e10b8cb04ee86bdf731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a5eba96a354e10b8cb04ee86bdf731" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43643 } } }
I20260620 06:11:49.226715 19847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35a5eba96a354e10b8cb04ee86bdf731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a5eba96a354e10b8cb04ee86bdf731" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43643 } } }
I20260620 06:11:49.226801 19847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:49.226801 19848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:49.227222 19855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:49.229982 19855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:49.235008 19855 catalog_manager.cc:1347] Generated new cluster ID: 72546623c7f84e2cb1ba86fef2cb4b10
I20260620 06:11:49.235090 19855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:49.256850 19855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:49.257144 19855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:49.263315 19855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731: Generated new TSK 0
I20260620 06:11:49.263957 19855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:49.270159 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43643
--builtin_ntp_servers=127.14.209.148:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:11:49.376106 19865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:49.376320 19865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:49.376356 19865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:49.380259 19865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:49.380306 19865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:49.380384 19865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:49.384984 19865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:49.385886 19865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:49.387179 19865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:49.396140 19870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.396136 19871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.396137 19873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:49.396543 19865 server_base.cc:1034] running on GCE node
I20260620 06:11:49.396867 19865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:49.397444 19865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:49.398604 19865 hybrid_clock.cc:648] HybridClock initialized: now 1781935909398586 us; error 29 us; skew 500 ppm
I20260620 06:11:49.400286 19865 webserver.cc:460] Webserver started at http://127.14.209.129:37931/ using document root <none> and password file <none>
I20260620 06:11:49.400818 19865 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:49.400878 19865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:49.401103 19865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:49.402964 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "8b6ed8ef87fb49c98d2825c51013d2ac"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "92cc0ceb72dd5d2542ead4a47e21dcd1"
server_key_iv: "147bd14668e58bab0a111d4dbbcb79f6"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.403553 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "8b6ed8ef87fb49c98d2825c51013d2ac"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "92cc0ceb72dd5d2542ead4a47e21dcd1"
server_key_iv: "147bd14668e58bab0a111d4dbbcb79f6"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.406859 19865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:11:49.409180 19879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.410300 19865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:49.410452 19865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "8b6ed8ef87fb49c98d2825c51013d2ac"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "92cc0ceb72dd5d2542ead4a47e21dcd1"
server_key_iv: "147bd14668e58bab0a111d4dbbcb79f6"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.410578 19865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:49.414767 19865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:49.415402 19865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:49.415575 19865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:49.416090 19865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:49.416911 19865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:49.416978 19865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.417037 19865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:49.417078 19865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.425444 19865 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:39407
I20260620 06:11:49.425485 19992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:39407 every 8 connection(s)
I20260620 06:11:49.426309 19865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:49.432255 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.428123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:48377 (local address 127.14.209.190:43643)
0620 06:11:49.428264 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.428268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.429210 (+ 942us) server_negotiation.cc:407] Connection header received
0620 06:11:49.430477 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.430480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.430527 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.430611 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.431786 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.431793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.431797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.431809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.431828 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.431837 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.431951 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.432085 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.432126 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:49.432987 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.428469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:43643 (local address 127.14.209.129:48377)
0620 06:11:49.429072 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.429108 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:49.430269 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:49.430775 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:49.430781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:49.430871 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:49.431482 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:49.431484 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:49.431519 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:49.431588 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:49.431935 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:49.431944 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:49.431949 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:11:49.432123 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 06:11:49.432334 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260620 06:11:49.434101 19993 heartbeater.cc:347] Connected to a master server at 127.14.209.190:43643
I20260620 06:11:49.434284 19993 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:49.434608 19993 heartbeater.cc:511] Master 127.14.209.190:43643 requested a full tablet report, sending...
I20260620 06:11:49.435961 19805 ts_manager.cc:194] Registered new tserver with Master: 8b6ed8ef87fb49c98d2825c51013d2ac (127.14.209.129:39407)
I20260620 06:11:49.436172 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19865
I20260620 06:11:49.436309 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:49.436575 15174 external_mini_cluster.cc:1428] Setting key b8e626c158f7770f68c0fe8e540bf6fb
I20260620 06:11:49.438184 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43643
--builtin_ntp_servers=127.14.209.148:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:11:49.542933 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:49.543205 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:49.543246 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:49.547178 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:49.547221 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:49.547273 19996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:49.551717 19996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:49.552544 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:49.553779 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:49.562564 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.562574 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:49.562619 19996 server_base.cc:1034] running on GCE node
W20260620 06:11:49.562583 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:49.563208 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:49.563799 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:49.564982 19996 hybrid_clock.cc:648] HybridClock initialized: now 1781935909564969 us; error 30 us; skew 500 ppm
I20260620 06:11:49.566670 19996 webserver.cc:460] Webserver started at http://127.14.209.130:39873/ using document root <none> and password file <none>
I20260620 06:11:49.567306 19996 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:49.567394 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:49.567595 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:49.569471 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "a8007ab113bd484a974a345e14800d62"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "bfd7abea01647337224963256baf8615"
server_key_iv: "bea876613798b5bae3a250cacabd9b9d"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.570004 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "a8007ab113bd484a974a345e14800d62"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "bfd7abea01647337224963256baf8615"
server_key_iv: "bea876613798b5bae3a250cacabd9b9d"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.573370 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:49.575753 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.576853 19996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:49.576987 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "a8007ab113bd484a974a345e14800d62"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "bfd7abea01647337224963256baf8615"
server_key_iv: "bea876613798b5bae3a250cacabd9b9d"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.577100 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:49.581305 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:49.581840 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:49.582015 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:49.582592 19996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:49.583503 19996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:49.583577 19996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.583642 19996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:49.583685 19996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.592247 19996 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:40401
I20260620 06:11:49.592267 20123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:40401 every 8 connection(s)
I20260620 06:11:49.593185 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:49.593585 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 19996
I20260620 06:11:49.593709 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:49.594008 15174 external_mini_cluster.cc:1428] Setting key 95fd81c02b4e591d0863490f4185ac3f
I20260620 06:11:49.596081 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43643
--builtin_ntp_servers=127.14.209.148:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 06:11:49.599545 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.594849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:33441 (local address 127.14.209.190:43643)
0620 06:11:49.594975 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.594986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.596226 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 06:11:49.597820 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.597823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.597857 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.597932 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.599126 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.599136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.599138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.599149 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.599167 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.599175 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.599249 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.599415 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.599450 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:11:49.600626 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.595369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:43643 (local address 127.14.209.130:33441)
0620 06:11:49.596084 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.596133 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:49.597601 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:49.598107 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:49.598117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:49.598223 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:49.598788 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:49.598792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:49.598852 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:49.598932 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:49.599264 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:49.599275 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:49.599284 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:11:49.599449 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 06:11:49.599670 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
I20260620 06:11:49.601750 20124 heartbeater.cc:347] Connected to a master server at 127.14.209.190:43643
I20260620 06:11:49.601953 20124 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:49.602304 20124 heartbeater.cc:511] Master 127.14.209.190:43643 requested a full tablet report, sending...
I20260620 06:11:49.603282 19805 ts_manager.cc:194] Registered new tserver with Master: a8007ab113bd484a974a345e14800d62 (127.14.209.130:40401)
W20260620 06:11:49.703303 20127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:49.703562 20127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:49.703639 20127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:49.707579 20127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:49.707648 20127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:49.707732 20127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:49.712520 20127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:49.713538 20127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:49.714916 20127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:49.724268 20135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.724262 20132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:49.724270 20133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:49.724792 20127 server_base.cc:1034] running on GCE node
I20260620 06:11:49.725104 20127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:49.725687 20127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:49.726898 20127 hybrid_clock.cc:648] HybridClock initialized: now 1781935909726885 us; error 28 us; skew 500 ppm
I20260620 06:11:49.728729 20127 webserver.cc:460] Webserver started at http://127.14.209.131:45719/ using document root <none> and password file <none>
I20260620 06:11:49.729291 20127 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:49.729383 20127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:49.729593 20127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:49.731592 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "4b73d6cd213744518123c173996c822b"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "afcd817bd5d74d2faffa869eb16de7b5"
server_key_iv: "aa11b575a4b1cf8836c446ca9404a149"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.732151 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "4b73d6cd213744518123c173996c822b"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "afcd817bd5d74d2faffa869eb16de7b5"
server_key_iv: "aa11b575a4b1cf8836c446ca9404a149"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.736626 20127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:11:49.738972 20141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.740140 20127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:49.740276 20127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "4b73d6cd213744518123c173996c822b"
format_stamp: "Formatted at 2026-06-20 06:11:49 on dist-test-slave-zt01"
server_key: "afcd817bd5d74d2faffa869eb16de7b5"
server_key_iv: "aa11b575a4b1cf8836c446ca9404a149"
server_key_version: "encryptionkey@0"
I20260620 06:11:49.740384 20127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:49.744741 20127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:49.745290 20127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:49.745474 20127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:49.745990 20127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:49.746812 20127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:49.746882 20127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.746945 20127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:49.746994 20127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:49.755584 20127 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:34397
I20260620 06:11:49.755676 20254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:34397 every 8 connection(s)
I20260620 06:11:49.756408 20127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:49.762181 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20127
I20260620 06:11:49.762308 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:49.762549 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.758180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:46445 (local address 127.14.209.190:43643)
0620 06:11:49.758326 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.758330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.759452 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 06:11:49.760817 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.760821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.760867 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.760991 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.761992 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.761996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.762000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.762013 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.762041 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.762049 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.762148 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.762349 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.762397 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:11:49.762591 15174 external_mini_cluster.cc:1428] Setting key 85e7ab51fffd670585d0acb49b47cd9f
I20260620 06:11:49.763756 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.758538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:43643 (local address 127.14.209.131:46445)
0620 06:11:49.759300 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.759350 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:49.760605 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:49.761147 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:49.761153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:49.761236 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:49.761733 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:49.761735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:49.761772 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:49.761835 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:49.762146 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:49.762155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:49.762163 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:11:49.762380 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 06:11:49.762604 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260620 06:11:49.764977 20255 heartbeater.cc:347] Connected to a master server at 127.14.209.190:43643
I20260620 06:11:49.765199 20255 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:49.765560 20255 heartbeater.cc:511] Master 127.14.209.190:43643 requested a full tablet report, sending...
I20260620 06:11:49.766496 19805 ts_manager.cc:194] Registered new tserver with Master: 4b73d6cd213744518123c173996c822b (127.14.209.131:34397)
I20260620 06:11:49.776608 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:11:49.780207 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.778480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.14.209.190:43643)
0620 06:11:49.778635 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.778639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.778844 (+ 205us) server_negotiation.cc:407] Connection header received
0620 06:11:49.778983 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.778986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.779028 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.779146 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.779806 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.779810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.779813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.779825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.779843 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.779852 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.779968 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.780039 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.780082 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:11:49.782969 19805 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:49.785972 19805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:49.797273 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.794407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39407 (local address 127.0.0.1:34788)
0620 06:11:49.794736 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.794749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:49.794821 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:49.795686 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:49.795689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:49.795705 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:49.796428 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:49.796430 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:49.796450 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:49.796546 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:49.796916 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:49.796925 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:49.796930 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:11:49.797035 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 06:11:49.797102 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":93,"threads_started":1}
I20260620 06:11:49.797324 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.793801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:34397 (local address 127.0.0.1:57396)
0620 06:11:49.794366 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.794411 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:49.794525 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:49.795570 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:49.795577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:49.795650 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:49.796623 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:49.796627 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:49.796640 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:49.796664 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:49.797122 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:49.797125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:49.797128 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:11:49.797182 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 06:11:49.797210 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":67,"threads_started":1}
I20260620 06:11:49.797801 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.794818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.14.209.129:39407)
0620 06:11:49.795297 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.795308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.795328 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:11:49.795376 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.795380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.795533 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.795920 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.796702 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.796709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.796722 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.796735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.796783 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.796813 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.796936 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.797126 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.797193 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":148,"threads_started":1}
I20260620 06:11:49.797848 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.793968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.14.209.131:34397)
0620 06:11:49.795100 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.795112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.795136 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:11:49.795185 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.795191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.795372 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.796303 (+ 931us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.796921 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.796926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.796935 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.796946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.796984 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.797011 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.797538 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.797587 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.797644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":132,"threads_started":1}
I20260620 06:11:49.797847 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.794518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.14.209.130:40401)
0620 06:11:49.795127 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:49.795139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:49.795164 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:11:49.795220 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:49.795226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:49.795405 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:49.795569 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:49.796801 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:49.796807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:49.796818 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:49.796837 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:49.796904 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:49.796939 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:49.797043 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:49.797390 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 06:11:49.797626 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":136,"threads_started":1}
I20260620 06:11:49.797325 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:49.794039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:40401 (local address 127.0.0.1:57338)
0620 06:11:49.794542 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:49.794557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:49.794640 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:49.795570 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:49.795577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:49.795650 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:49.796657 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:49.796659 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:49.796666 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:49.796684 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:49.797023 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:49.797030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:49.797037 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:11:49.797127 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:11:49.797169 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":89,"threads_started":1}
I20260620 06:11:49.800443 20189 tablet_service.cc:1467] Processing CreateTablet for tablet f9faa2f0f6a84b5797944bab0426bb99 (DEFAULT_TABLE table=test-table [id=a8d84ddc35dd4d6da6aeaef3ab65f77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:49.800802 19927 tablet_service.cc:1467] Processing CreateTablet for tablet f9faa2f0f6a84b5797944bab0426bb99 (DEFAULT_TABLE table=test-table [id=a8d84ddc35dd4d6da6aeaef3ab65f77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:49.800803 20058 tablet_service.cc:1467] Processing CreateTablet for tablet f9faa2f0f6a84b5797944bab0426bb99 (DEFAULT_TABLE table=test-table [id=a8d84ddc35dd4d6da6aeaef3ab65f77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:49.801492 20189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9faa2f0f6a84b5797944bab0426bb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:49.801792 20058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9faa2f0f6a84b5797944bab0426bb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:49.801792 19927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9faa2f0f6a84b5797944bab0426bb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:49.807194 20272 tablet_bootstrap.cc:492] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Bootstrap starting.
I20260620 06:11:49.808867 20273 tablet_bootstrap.cc:492] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Bootstrap starting.
I20260620 06:11:49.810065 20272 tablet_bootstrap.cc:654] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:49.810995 20273 tablet_bootstrap.cc:654] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:49.810974 20272 log.cc:824] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:49.811379 20274 tablet_bootstrap.cc:492] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: Bootstrap starting.
I20260620 06:11:49.811758 20273 log.cc:824] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:49.812995 20272 tablet_bootstrap.cc:492] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: No bootstrap required, opened a new log
I20260620 06:11:49.813184 20272 ts_tablet_manager.cc:1397] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:11:49.813380 20274 tablet_bootstrap.cc:654] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:49.813622 20273 tablet_bootstrap.cc:492] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: No bootstrap required, opened a new log
I20260620 06:11:49.813786 20273 ts_tablet_manager.cc:1397] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:11:49.814116 20274 log.cc:824] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:49.815966 20274 tablet_bootstrap.cc:492] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: No bootstrap required, opened a new log
I20260620 06:11:49.816154 20274 ts_tablet_manager.cc:1397] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:11:49.817744 20272 raft_consensus.cc:348] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:49.817992 20272 raft_consensus.cc:374] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:49.818055 20272 raft_consensus.cc:729] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b73d6cd213744518123c173996c822b, State: Initialized, Role: FOLLOWER
I20260620 06:11:49.818069 20273 raft_consensus.cc:348] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:49.818287 20273 raft_consensus.cc:374] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:49.818337 20273 raft_consensus.cc:729] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8007ab113bd484a974a345e14800d62, State: Initialized, Role: FOLLOWER
I20260620 06:11:49.818495 20272 consensus_queue.cc:260] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:49.818748 20273 consensus_queue.cc:260] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:49.819298 20255 heartbeater.cc:503] Master 127.14.209.190:43643 was elected leader, sending a full tablet report...
I20260620 06:11:49.819512 20272 ts_tablet_manager.cc:1428] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260620 06:11:49.819535 20273 ts_tablet_manager.cc:1428] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:11:49.819633 20274 raft_consensus.cc:348] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:49.819808 20124 heartbeater.cc:503] Master 127.14.209.190:43643 was elected leader, sending a full tablet report...
I20260620 06:11:49.819834 20274 raft_consensus.cc:374] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:49.819877 20274 raft_consensus.cc:729] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6ed8ef87fb49c98d2825c51013d2ac, State: Initialized, Role: FOLLOWER
I20260620 06:11:49.820264 20274 consensus_queue.cc:260] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:49.820871 19993 heartbeater.cc:503] Master 127.14.209.190:43643 was elected leader, sending a full tablet report...
I20260620 06:11:49.822718 20274 ts_tablet_manager.cc:1428] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 06:11:49.844198 20125 tablet.cc:2369] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:11:49.927443 19994 tablet.cc:2369] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:11:50.007304 20256 tablet.cc:2369] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:50.071749 20279 raft_consensus.cc:482] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:50.071954 20279 raft_consensus.cc:504] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:50.073192 20279 leader_election.cc:288] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b73d6cd213744518123c173996c822b (127.14.209.131:34397), 8b6ed8ef87fb49c98d2825c51013d2ac (127.14.209.129:39407)
I20260620 06:11:50.075309 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.073510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39407 (local address 127.14.209.130:33601)
0620 06:11:50.073652 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:50.073669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:50.073759 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:50.074099 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:50.074105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:50.074123 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:50.074710 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:50.074711 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:50.074721 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:50.074744 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:50.075024 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:50.075027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:50.075030 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:11:50.075146 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 06:11:50.075185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:11:50.075309 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.073594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:33601 (local address 127.14.209.129:39407)
0620 06:11:50.073746 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.073750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.073769 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:11:50.073864 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.073896 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.073940 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.074040 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:50.074885 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:50.074891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:50.074898 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:50.074909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:50.074928 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:50.074936 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:50.075030 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:50.075155 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 06:11:50.075192 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:11:50.075733 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.073713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:57505 (local address 127.14.209.131:34397)
0620 06:11:50.073845 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.073851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.074069 (+ 218us) server_negotiation.cc:407] Connection header received
0620 06:11:50.074140 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.074143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.074184 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.074275 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:50.075125 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:50.075128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:50.075131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:50.075150 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:50.075169 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:50.075176 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:50.075294 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:50.075575 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 06:11:50.075616 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:50.075763 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.073527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:34397 (local address 127.14.209.130:57505)
0620 06:11:50.073958 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:50.073973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:50.074060 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:50.074361 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:50.074366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:50.074402 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:50.074911 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:11:50.074913 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:50.074922 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:11:50.074946 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:50.075278 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:50.075281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:50.075284 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:11:50.075573 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 06:11:50.075612 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":147,"threads_started":1}
I20260620 06:11:50.076051 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9faa2f0f6a84b5797944bab0426bb99" candidate_uuid: "a8007ab113bd484a974a345e14800d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" is_pre_election: true
I20260620 06:11:50.076347 19947 raft_consensus.cc:2454] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8007ab113bd484a974a345e14800d62 in term 0.
I20260620 06:11:50.076428 20209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9faa2f0f6a84b5797944bab0426bb99" candidate_uuid: "a8007ab113bd484a974a345e14800d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b73d6cd213744518123c173996c822b" is_pre_election: true
I20260620 06:11:50.076725 20209 raft_consensus.cc:2454] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8007ab113bd484a974a345e14800d62 in term 0.
I20260620 06:11:50.076846 20014 leader_election.cc:302] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6ed8ef87fb49c98d2825c51013d2ac, a8007ab113bd484a974a345e14800d62; no voters:
I20260620 06:11:50.077106 20279 raft_consensus.cc:2781] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:50.077193 20279 raft_consensus.cc:482] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:50.077251 20279 raft_consensus.cc:3037] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:50.078809 20279 raft_consensus.cc:504] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:50.079298 20279 leader_election.cc:288] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [CANDIDATE]: Term 1 election: Requested vote from peers 4b73d6cd213744518123c173996c822b (127.14.209.131:34397), 8b6ed8ef87fb49c98d2825c51013d2ac (127.14.209.129:39407)
I20260620 06:11:50.079622 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9faa2f0f6a84b5797944bab0426bb99" candidate_uuid: "a8007ab113bd484a974a345e14800d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac"
I20260620 06:11:50.079731 20209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9faa2f0f6a84b5797944bab0426bb99" candidate_uuid: "a8007ab113bd484a974a345e14800d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b73d6cd213744518123c173996c822b"
I20260620 06:11:50.079932 19947 raft_consensus.cc:3037] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:50.079932 20209 raft_consensus.cc:3037] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:50.081180 19947 raft_consensus.cc:2454] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8007ab113bd484a974a345e14800d62 in term 1.
I20260620 06:11:50.081168 20209 raft_consensus.cc:2454] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8007ab113bd484a974a345e14800d62 in term 1.
I20260620 06:11:50.081480 20014 leader_election.cc:302] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6ed8ef87fb49c98d2825c51013d2ac, a8007ab113bd484a974a345e14800d62; no voters:
I20260620 06:11:50.081677 20279 raft_consensus.cc:2781] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:50.081915 20279 raft_consensus.cc:686] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 1 LEADER]: Becoming Leader. State: Replica: a8007ab113bd484a974a345e14800d62, State: Running, Role: LEADER
I20260620 06:11:50.082252 20279 consensus_queue.cc:237] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } }
I20260620 06:11:50.085661 19801 catalog_manager.cc:5495] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8007ab113bd484a974a345e14800d62 (127.14.209.130). New cstate: current_term: 1 leader_uuid: "a8007ab113bd484a974a345e14800d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8007ab113bd484a974a345e14800d62" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 } health_report { overall_health: UNKNOWN } } }
I20260620 06:11:50.159006 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.157459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.14.209.130:40401)
0620 06:11:50.157610 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.157614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.157630 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:50.157727 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.157729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.157785 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.157875 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:50.158597 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:50.158600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:50.158603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:50.158614 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:50.158637 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:50.158645 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:50.158736 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:50.158824 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 06:11:50.158863 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:11:50.164537 20209 raft_consensus.cc:1261] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 1 FOLLOWER]: Refusing update from remote peer a8007ab113bd484a974a345e14800d62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:50.164534 19947 raft_consensus.cc:1261] T f9faa2f0f6a84b5797944bab0426bb99 P 8b6ed8ef87fb49c98d2825c51013d2ac [term 1 FOLLOWER]: Refusing update from remote peer a8007ab113bd484a974a345e14800d62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:50.165187 20279 consensus_queue.cc:1035] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b73d6cd213744518123c173996c822b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:50.165424 20282 consensus_queue.cc:1035] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6ed8ef87fb49c98d2825c51013d2ac" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:50.173659 20287 mvcc.cc:205] Tried to move back new op lower bound from 7298809488027176960 to 7298809487699292160. Current Snapshot: MvccSnapshot[applied={T|T < 7298809488027176960}]
I20260620 06:11:50.180644 19801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:50.180810 19801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:50.183115 19801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35a5eba96a354e10b8cb04ee86bdf731: Sending DeleteTablet for 3 replicas of tablet f9faa2f0f6a84b5797944bab0426bb99
I20260620 06:11:50.183823 20189 tablet_service.cc:1514] Processing DeleteTablet for tablet f9faa2f0f6a84b5797944bab0426bb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:50 UTC) from {username='slave'} at 127.0.0.1:57396
I20260620 06:11:50.184064 19927 tablet_service.cc:1514] Processing DeleteTablet for tablet f9faa2f0f6a84b5797944bab0426bb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:50 UTC) from {username='slave'} at 127.0.0.1:34788
I20260620 06:11:50.184276 20295 tablet_replica.cc:330] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: stopping tablet replica
I20260620 06:11:50.184303 20058 tablet_service.cc:1514] Processing DeleteTablet for tablet f9faa2f0f6a84b5797944bab0426bb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:50 UTC) from {username='slave'} at 127.0.0.1:57338
I20260620 06:11:50.184585 20295 raft_consensus.cc:2229] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:50.184725 20297 tablet_replica.cc:330] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: stopping tablet replica
I20260620 06:11:50.184885 20295 raft_consensus.cc:2258] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:50.184983 20297 raft_consensus.cc:2229] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:50.185289 20297 raft_consensus.cc:2258] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:50.185444 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19865
I20260620 06:11:50.186923 20297 ts_tablet_manager.cc:1905] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:50.186944 20295 ts_tablet_manager.cc:1905] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:50.189683 20297 ts_tablet_manager.cc:1918] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:50.189776 20297 log.cc:1196] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/f9faa2f0f6a84b5797944bab0426bb99
I20260620 06:11:50.190081 20297 ts_tablet_manager.cc:1939] T f9faa2f0f6a84b5797944bab0426bb99 P a8007ab113bd484a974a345e14800d62: Deleting consensus metadata
I20260620 06:11:50.190222 20295 ts_tablet_manager.cc:1918] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:50.190327 20295 log.cc:1196] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/f9faa2f0f6a84b5797944bab0426bb99
I20260620 06:11:50.190778 20295 ts_tablet_manager.cc:1939] T f9faa2f0f6a84b5797944bab0426bb99 P 4b73d6cd213744518123c173996c822b: Deleting consensus metadata
I20260620 06:11:50.191193 19791 catalog_manager.cc:4841] TS a8007ab113bd484a974a345e14800d62 (127.14.209.130:40401): tablet f9faa2f0f6a84b5797944bab0426bb99 (table test-table [id=a8d84ddc35dd4d6da6aeaef3ab65f77c]) successfully deleted
I20260620 06:11:50.191915 19789 catalog_manager.cc:4841] TS 4b73d6cd213744518123c173996c822b (127.14.209.131:34397): tablet f9faa2f0f6a84b5797944bab0426bb99 (table test-table [id=a8d84ddc35dd4d6da6aeaef3ab65f77c]) successfully deleted
W20260620 06:11:50.194393 19792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:39407 (error 108)
I20260620 06:11:50.194563 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19996
I20260620 06:11:50.196849 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.196602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39407 (local address 127.0.0.1:34796)
0620 06:11:50.196717 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:50.196776 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:11:50.197070 19792 catalog_manager.cc:4584] TS 8b6ed8ef87fb49c98d2825c51013d2ac (127.14.209.129:39407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9faa2f0f6a84b5797944bab0426bb99: Network error: Client connection negotiation failed: client connection to 127.14.209.129:39407: connect: Connection refused (error 111)
I20260620 06:11:50.199395 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20127
I20260620 06:11:50.204610 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 19774
2026-06-20T06:11:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1185 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 06:11:50.226006 15174 mini_oidc.cc:152] Starting JWKS server
I20260620 06:11:50.227738 15174 webserver.cc:460] Webserver started at https://127.0.0.1:36073/ using document root <none> and password file <none>
I20260620 06:11:50.227866 15174 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:11:50.228088 15174 webserver.cc:460] Webserver started at http://127.0.0.1:39391/ using document root <none> and password file <none>
2026-06-20T06:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:50Z Disabled control of system clock
I20260620 06:11:50.236510 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:44013
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:44013
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36073/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/testchainca.cert with env {}
W20260620 06:11:50.343163 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:50.343397 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:50.343434 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:50.347295 20304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:50.347342 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:50.347357 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:50.347373 20304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:50.347388 20304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:11:50.347404 20304 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:11:50.347416 20304 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36073/jwks.json
I20260620 06:11:50.352494 20304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:44013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:44013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36073/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:50.353374 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:50.354259 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:50.362223 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:50.362241 20304 server_base.cc:1034] running on GCE node
W20260620 06:11:50.362258 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:50.362248 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:50.363008 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:50.364043 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:50.365208 20304 hybrid_clock.cc:648] HybridClock initialized: now 1781935910365181 us; error 41 us; skew 500 ppm
I20260620 06:11:50.366853 20304 webserver.cc:460] Webserver started at http://127.14.209.190:39967/ using document root <none> and password file <none>
I20260620 06:11:50.367403 20304 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:50.367461 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:50.367614 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:50.369398 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "ef326e7649984d9bbfa89f09147a3855"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "46ac7b99957da2df6255c052ae448a37"
server_key_iv: "760714dab3ce5592d802ddf2f003ae76"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.369879 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "ef326e7649984d9bbfa89f09147a3855"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "46ac7b99957da2df6255c052ae448a37"
server_key_iv: "760714dab3ce5592d802ddf2f003ae76"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.373201 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 06:11:50.375407 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:50.376549 20304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:50.376642 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "ef326e7649984d9bbfa89f09147a3855"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "46ac7b99957da2df6255c052ae448a37"
server_key_iv: "760714dab3ce5592d802ddf2f003ae76"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.376719 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:50.399827 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:50.400424 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:50.400570 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:50.407250 20304 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:44013
I20260620 06:11:50.407322 20370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:44013 every 8 connection(s)
I20260620 06:11:50.408114 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:50.411026 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:50.411890 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20304
I20260620 06:11:50.412026 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:50.412309 15174 external_mini_cluster.cc:1428] Setting key 6c8651b3bf5788f5487fea78846ea01d
I20260620 06:11:50.417729 20371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855: Bootstrap starting.
I20260620 06:11:50.419953 20371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:50.420733 20371 log.cc:824] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:50.421001 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.413469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.14.209.190:44013)
0620 06:11:50.414053 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.414069 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.414101 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:11:50.414749 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.414764 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.415220 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.415539 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:50.415937 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.416833 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.417706 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.418196 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.418357 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:50.419343 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:50.419348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:50.419359 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:50.419715 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:50.419767 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:50.419786 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:50.419895 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:50.420068 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 06:11:50.420299 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260620 06:11:50.422441 20371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855: No bootstrap required, opened a new log
I20260620 06:11:50.423560 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:50.426146 20371 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef326e7649984d9bbfa89f09147a3855" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44013 } }
I20260620 06:11:50.426354 20371 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:50.426395 20371 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef326e7649984d9bbfa89f09147a3855, State: Initialized, Role: FOLLOWER
I20260620 06:11:50.426734 20371 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef326e7649984d9bbfa89f09147a3855" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44013 } }
I20260620 06:11:50.426836 20371 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:50.426872 20371 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:50.426934 20371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:50.427740 20371 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef326e7649984d9bbfa89f09147a3855" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44013 } }
I20260620 06:11:50.428051 20371 leader_election.cc:302] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef326e7649984d9bbfa89f09147a3855; no voters:
I20260620 06:11:50.428298 20371 leader_election.cc:288] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:50.428463 20376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:50.428710 20376 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [term 1 LEADER]: Becoming Leader. State: Replica: ef326e7649984d9bbfa89f09147a3855, State: Running, Role: LEADER
I20260620 06:11:50.429015 20376 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef326e7649984d9bbfa89f09147a3855" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44013 } }
I20260620 06:11:50.429503 20371 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:50.430575 20377 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef326e7649984d9bbfa89f09147a3855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef326e7649984d9bbfa89f09147a3855" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44013 } } }
I20260620 06:11:50.430614 20378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef326e7649984d9bbfa89f09147a3855. Latest consensus state: current_term: 1 leader_uuid: "ef326e7649984d9bbfa89f09147a3855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef326e7649984d9bbfa89f09147a3855" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 44013 } } }
I20260620 06:11:50.430699 20377 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:50.430711 20378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:50.431033 20385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:50.433995 20385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:50.439416 20385 catalog_manager.cc:1347] Generated new cluster ID: 195792d331944f5aadd772b0094cda3c
I20260620 06:11:50.439468 20385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:50.450560 20385 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:50.451697 20385 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:50.461980 20385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef326e7649984d9bbfa89f09147a3855: Generated new TSK 0
I20260620 06:11:50.462738 20385 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:11:50.497963 20401 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 06:11:50.498082 20401 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:11:50.498553 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.482036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.14.209.190:44013)
0620 06:11:50.482180 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.482185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.482437 (+ 252us) server_negotiation.cc:407] Connection header received
0620 06:11:50.482639 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.482641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.482687 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.482768 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:11:50.483046 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.483648 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.484860 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.485079 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.485149 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:50.485491 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:11:50.498247 (+ 12756us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:11:50.498450 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 06:11:50.498633 20374 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 06:11:50.499930 20400 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:50.481976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:44013 (local address 127.0.0.1:46880)
0620 06:11:50.482340 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:50.482355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:50.482491 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:50.482811 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:50.482814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:50.482827 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:11:50.482944 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:50.482950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.483764 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.483768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:50.484741 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:50.484751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.485183 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.485187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:50.485279 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:50.485326 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:11:50.498448 (+ 13122us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:11:50.498672 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.209.190:44013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":146,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.209.190:44013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 06:11:50.501008 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20304
2026-06-20T06:11:50Z chronyd exiting
I20260620 06:11:50.538713 15174 test_util.cc:175] -----------------------------------------------
I20260620 06:11:50.538812 15174 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935885709390-15174-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 06:11:50.539945 15174 mini_oidc.cc:152] Starting JWKS server
I20260620 06:11:50.540828 15174 webserver.cc:460] Webserver started at https://127.0.0.1:39421/ using document root <none> and password file <none>
I20260620 06:11:50.540957 15174 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:11:50.541215 15174 webserver.cc:460] Webserver started at http://127.0.0.1:43701/ using document root <none> and password file <none>
2026-06-20T06:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:50Z Disabled control of system clock
I20260620 06:11:50.549832 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39275
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:39275
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/testchainca.cert with env {}
W20260620 06:11:50.653404 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:50.653646 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:50.653683 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:50.657604 20407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:50.657657 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:50.657673 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:50.657689 20407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:50.657704 20407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:11:50.657720 20407 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:11:50.657734 20407 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39421/jwks.json
I20260620 06:11:50.662741 20407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:39275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:50.663650 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:50.664608 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:50.673308 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:50.673293 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:50.673451 20407 server_base.cc:1034] running on GCE node
W20260620 06:11:50.673297 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:50.674196 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:50.675318 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:50.676523 20407 hybrid_clock.cc:648] HybridClock initialized: now 1781935910676509 us; error 27 us; skew 500 ppm
I20260620 06:11:50.678377 20407 webserver.cc:460] Webserver started at http://127.14.209.190:34687/ using document root <none> and password file <none>
I20260620 06:11:50.678934 20407 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:50.679028 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:50.679276 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:50.681200 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "8b027e68e04e4249b69736a699fa4b3c"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "cbbb9d127758456be264b465c26a1371"
server_key_iv: "97ab44d5e54ab1d357e58239430146f4"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.681743 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "8b027e68e04e4249b69736a699fa4b3c"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "cbbb9d127758456be264b465c26a1371"
server_key_iv: "97ab44d5e54ab1d357e58239430146f4"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.685267 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:50.687572 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:50.688687 20407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:50.688825 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "8b027e68e04e4249b69736a699fa4b3c"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "cbbb9d127758456be264b465c26a1371"
server_key_iv: "97ab44d5e54ab1d357e58239430146f4"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.688932 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:50.700484 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:50.701045 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:50.701231 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:50.708019 20407 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:39275
I20260620 06:11:50.708043 20473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:39275 every 8 connection(s)
I20260620 06:11:50.709185 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:50.712602 20474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:50.714972 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20407
I20260620 06:11:50.715132 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:50.715448 15174 external_mini_cluster.cc:1428] Setting key e191b7385d726f41c84e9e4fe840395b
I20260620 06:11:50.718626 20474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c: Bootstrap starting.
I20260620 06:11:50.720968 20474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:50.721717 20474 log.cc:824] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:50.723766 20474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c: No bootstrap required, opened a new log
I20260620 06:11:50.724359 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.716784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.14.209.190:39275)
0620 06:11:50.717350 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.717364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.717396 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:11:50.718005 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.718020 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.718367 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.718631 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:50.719006 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.719972 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.721156 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.721605 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.721722 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:50.722690 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:50.722695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:50.722706 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:50.723099 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:50.723160 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:50.723179 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:50.723250 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:50.723405 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 06:11:50.723679 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260620 06:11:50.727254 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:50.727672 20474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b027e68e04e4249b69736a699fa4b3c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39275 } }
I20260620 06:11:50.727861 20474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:50.727926 20474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b027e68e04e4249b69736a699fa4b3c, State: Initialized, Role: FOLLOWER
I20260620 06:11:50.728384 20474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b027e68e04e4249b69736a699fa4b3c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39275 } }
I20260620 06:11:50.728539 20474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:50.728595 20474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:50.728688 20474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:50.729480 20474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b027e68e04e4249b69736a699fa4b3c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39275 } }
I20260620 06:11:50.729794 20474 leader_election.cc:302] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b027e68e04e4249b69736a699fa4b3c; no voters:
I20260620 06:11:50.730072 20474 leader_election.cc:288] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:50.730191 20479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:50.730466 20479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [term 1 LEADER]: Becoming Leader. State: Replica: 8b027e68e04e4249b69736a699fa4b3c, State: Running, Role: LEADER
I20260620 06:11:50.730741 20479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b027e68e04e4249b69736a699fa4b3c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39275 } }
I20260620 06:11:50.731256 20474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:50.732301 20481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b027e68e04e4249b69736a699fa4b3c. Latest consensus state: current_term: 1 leader_uuid: "8b027e68e04e4249b69736a699fa4b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b027e68e04e4249b69736a699fa4b3c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39275 } } }
I20260620 06:11:50.732442 20481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:50.733312 20488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:50.733404 20480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b027e68e04e4249b69736a699fa4b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b027e68e04e4249b69736a699fa4b3c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39275 } } }
I20260620 06:11:50.733484 20480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:50.735869 20488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:50.741485 20488 catalog_manager.cc:1347] Generated new cluster ID: 7313279a8826440ea58e1bd4745ef143
I20260620 06:11:50.741550 20488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:50.761823 20488 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:50.762645 20488 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:50.767292 20488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b027e68e04e4249b69736a699fa4b3c: Generated new TSK 0
I20260620 06:11:50.767956 20488 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:11:50.791368 20504 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 06:11:50.791438 20504 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:11:50.792174 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:50.776631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.14.209.190:39275)
0620 06:11:50.776759 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:50.776763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:50.776960 (+ 197us) server_negotiation.cc:407] Connection header received
0620 06:11:50.777119 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:50.777121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:50.777164 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:50.777241 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:11:50.777490 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.777998 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.779238 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.779449 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.779529 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:50.779851 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:11:50.791697 (+ 11846us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:11:50.792051 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 06:11:50.792202 20503 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:50.776556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:39275 (local address 127.0.0.1:34382)
0620 06:11:50.776870 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:50.776885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:50.776993 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:50.777271 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:50.777274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:50.777282 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:11:50.777383 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:50.777389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.778115 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.778120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:50.779129 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:50.779139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:50.779573 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:50.779577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:50.779669 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:50.779695 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:11:50.791849 (+ 12154us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:11:50.792046 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.209.190:39275: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
W20260620 06:11:50.792237 20477 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 06:11:50.792860 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20407
2026-06-20T06:11:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (291 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 06:11:50.831658 15174 mini_oidc.cc:152] Starting JWKS server
I20260620 06:11:50.832527 15174 webserver.cc:460] Webserver started at https://127.0.0.1:41523/ using document root <none> and password file <none>
I20260620 06:11:50.832634 15174 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:11:50.832917 15174 webserver.cc:460] Webserver started at http://127.0.0.1:45673/ using document root <none> and password file <none>
2026-06-20T06:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:50Z Disabled control of system clock
I20260620 06:11:50.841536 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40905
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41523/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:11:50.946952 20510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:50.947206 20510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:50.947243 20510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:50.951126 20510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:50.951174 20510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:50.951189 20510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:50.951205 20510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:50.951220 20510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:11:50.951313 20510 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:11:50.951377 20510 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41523/jwks.json
I20260620 06:11:50.956158 20510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41523/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:50.957134 20510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:50.958438 20510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:50.966648 20518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:50.966667 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:50.966727 20510 server_base.cc:1034] running on GCE node
W20260620 06:11:50.966667 20516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:50.967659 20510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:50.968711 20510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:50.969934 20510 hybrid_clock.cc:648] HybridClock initialized: now 1781935910969899 us; error 54 us; skew 500 ppm
I20260620 06:11:50.971742 20510 webserver.cc:460] Webserver started at http://127.14.209.190:42883/ using document root <none> and password file <none>
I20260620 06:11:50.972287 20510 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:50.972376 20510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:50.972594 20510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:50.974488 20510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "67065297571443858fa47e1ada3cac4f"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "28f58cb1acff1273db5d6fd49eac1097"
server_key_iv: "99bc024963b1419f82e4836529d7303e"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.975088 20510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "67065297571443858fa47e1ada3cac4f"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "28f58cb1acff1273db5d6fd49eac1097"
server_key_iv: "99bc024963b1419f82e4836529d7303e"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.978428 20510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:50.980681 20524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:50.981751 20510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 06:11:50.981895 20510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "67065297571443858fa47e1ada3cac4f"
format_stamp: "Formatted at 2026-06-20 06:11:50 on dist-test-slave-zt01"
server_key: "28f58cb1acff1273db5d6fd49eac1097"
server_key_iv: "99bc024963b1419f82e4836529d7303e"
server_key_version: "encryptionkey@0"
I20260620 06:11:50.982003 20510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:50.993366 20510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:50.993927 20510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:50.994107 20510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:51.000808 20510 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40905
I20260620 06:11:51.000833 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40905 every 8 connection(s)
I20260620 06:11:51.001627 20510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:51.004436 20577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:51.006611 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20510
I20260620 06:11:51.006740 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:51.007030 15174 external_mini_cluster.cc:1428] Setting key 02dfa69b86d53859f17745feb4863abd
I20260620 06:11:51.010093 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f: Bootstrap starting.
I20260620 06:11:51.012162 20577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:51.012840 20577 log.cc:824] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:51.014510 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f: No bootstrap required, opened a new log
I20260620 06:11:51.015825 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:51.008358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.14.209.190:40905)
0620 06:11:51.008932 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:51.008945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:51.008979 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:51.009597 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:51.009612 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:51.009951 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:51.010261 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:51.010651 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.011694 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.012619 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.013145 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.013314 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:51.014137 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:51.014142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:51.014154 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:51.014508 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:51.014559 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:51.014577 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:51.014696 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:51.014852 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 06:11:51.015158 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260620 06:11:51.018590 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:51.018494 20577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67065297571443858fa47e1ada3cac4f" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40905 } }
I20260620 06:11:51.018760 20577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:51.018811 20577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67065297571443858fa47e1ada3cac4f, State: Initialized, Role: FOLLOWER
I20260620 06:11:51.019260 20577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67065297571443858fa47e1ada3cac4f" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40905 } }
I20260620 06:11:51.019419 20577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:51.019517 20577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:51.019659 20577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:51.020625 20577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67065297571443858fa47e1ada3cac4f" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40905 } }
I20260620 06:11:51.020951 20577 leader_election.cc:302] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67065297571443858fa47e1ada3cac4f; no voters:
I20260620 06:11:51.021242 20577 leader_election.cc:288] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:51.021474 20582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:51.021732 20582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [term 1 LEADER]: Becoming Leader. State: Replica: 67065297571443858fa47e1ada3cac4f, State: Running, Role: LEADER
I20260620 06:11:51.022068 20582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67065297571443858fa47e1ada3cac4f" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40905 } }
I20260620 06:11:51.022428 20577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:51.023598 20584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67065297571443858fa47e1ada3cac4f. Latest consensus state: current_term: 1 leader_uuid: "67065297571443858fa47e1ada3cac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67065297571443858fa47e1ada3cac4f" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40905 } } }
I20260620 06:11:51.023727 20584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:51.024091 20583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67065297571443858fa47e1ada3cac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67065297571443858fa47e1ada3cac4f" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40905 } } }
I20260620 06:11:51.024180 20583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:51.026949 20597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:51.027034 20597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:51.027143 20592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:51.027783 20592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:51.033016 20592 catalog_manager.cc:1347] Generated new cluster ID: 2c37677fa18644249db0030e470f25f4
I20260620 06:11:51.033085 20592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:51.057104 20592 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:51.058328 20592 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:51.066438 20592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67065297571443858fa47e1ada3cac4f: Generated new TSK 0
I20260620 06:11:51.067224 20592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:51.087734 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:51.080423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.14.209.190:40905)
0620 06:11:51.080543 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:51.080548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:51.080687 (+ 139us) server_negotiation.cc:407] Connection header received
0620 06:11:51.080832 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:51.080834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:51.080880 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:51.080967 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:11:51.081306 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.081872 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.083180 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.083422 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.083508 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:51.083986 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:11:51.087204 (+ 3218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:11:51.087553 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 06:11:51.087837 20580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 06:11:51.090374 20606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:51.080278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40905 (local address 127.0.0.1:35360)
0620 06:11:51.080589 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:51.080603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:51.080713 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:51.080991 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:51.080995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:51.081003 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:11:51.081118 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:51.081125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.081985 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.081989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:51.083005 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:51.083015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.083557 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.083562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:51.083656 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:51.083685 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:11:51.087443 (+ 3758us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:11:51.090220 (+ 2777us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.209.190:40905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41523/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2508,"thread_start_us":103,"threads_started":1}
I20260620 06:11:51.091300 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20510
E20260620 06:11:51.100232 20607 webserver.cc:533] Webserver: sslize failed:
E20260620 06:11:51.100296 20607 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T06:11:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T06:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:51Z Disabled control of system clock
I20260620 06:11:51.138885 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39165
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:39165
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/testchainca.cert with env {}
W20260620 06:11:51.244122 20611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:51.244350 20611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:51.244387 20611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:51.248353 20611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:51.248400 20611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:51.248415 20611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:51.248432 20611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:51.248447 20611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:11:51.248464 20611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:11:51.248477 20611 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 06:11:51.253559 20611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:39165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:51.254413 20611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:51.255321 20611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:51.262995 20616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:51.263038 20617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:51.263038 20619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:51.263319 20611 server_base.cc:1034] running on GCE node
I20260620 06:11:51.263785 20611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:51.264729 20611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:51.265885 20611 hybrid_clock.cc:648] HybridClock initialized: now 1781935911265874 us; error 24 us; skew 500 ppm
I20260620 06:11:51.267584 20611 webserver.cc:460] Webserver started at http://127.14.209.190:42369/ using document root <none> and password file <none>
I20260620 06:11:51.268100 20611 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:51.268157 20611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:51.268373 20611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:51.270249 20611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "b23c7ec1cf3346698035f0e137e5f578"
format_stamp: "Formatted at 2026-06-20 06:11:51 on dist-test-slave-zt01"
server_key: "5acfc1cae26726ba2701e35105a4dd2e"
server_key_iv: "db93250b0998a924d6dbd84851b3ebee"
server_key_version: "encryptionkey@0"
I20260620 06:11:51.270879 20611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "b23c7ec1cf3346698035f0e137e5f578"
format_stamp: "Formatted at 2026-06-20 06:11:51 on dist-test-slave-zt01"
server_key: "5acfc1cae26726ba2701e35105a4dd2e"
server_key_iv: "db93250b0998a924d6dbd84851b3ebee"
server_key_version: "encryptionkey@0"
I20260620 06:11:51.274266 20611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:51.276587 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:51.277714 20611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:51.277854 20611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "b23c7ec1cf3346698035f0e137e5f578"
format_stamp: "Formatted at 2026-06-20 06:11:51 on dist-test-slave-zt01"
server_key: "5acfc1cae26726ba2701e35105a4dd2e"
server_key_iv: "db93250b0998a924d6dbd84851b3ebee"
server_key_version: "encryptionkey@0"
I20260620 06:11:51.277978 20611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:51.301836 20611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:51.302520 20611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:51.302721 20611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:51.310491 20611 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:39165
I20260620 06:11:51.310871 20677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:39165 every 8 connection(s)
I20260620 06:11:51.311576 20611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:51.314013 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20611
I20260620 06:11:51.314190 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:51.314489 15174 external_mini_cluster.cc:1428] Setting key 70e5ebe0c84d0c900d2bc97b2f8ef704
I20260620 06:11:51.314675 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:51.322628 20678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578: Bootstrap starting.
I20260620 06:11:51.323990 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:51.316083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.14.209.190:39165)
0620 06:11:51.316771 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:51.316786 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:51.316822 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:11:51.317598 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:51.317615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:51.318033 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:51.318370 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:51.318789 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.319841 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.320723 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.321187 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.321355 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:51.322296 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:51.322300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:51.322322 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:51.322715 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:51.322789 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:51.322811 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:51.322943 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:51.323087 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 06:11:51.323333 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":162,"threads_started":1}
I20260620 06:11:51.325094 20678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:51.325906 20678 log.cc:824] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:51.326825 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:51.328052 20678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578: No bootstrap required, opened a new log
I20260620 06:11:51.331789 20678 raft_consensus.cc:348] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23c7ec1cf3346698035f0e137e5f578" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39165 } }
I20260620 06:11:51.332057 20678 raft_consensus.cc:374] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:51.332139 20678 raft_consensus.cc:729] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23c7ec1cf3346698035f0e137e5f578, State: Initialized, Role: FOLLOWER
I20260620 06:11:51.332614 20678 consensus_queue.cc:260] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23c7ec1cf3346698035f0e137e5f578" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39165 } }
I20260620 06:11:51.332739 20678 raft_consensus.cc:388] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:51.332825 20678 raft_consensus.cc:482] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:51.332958 20678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:51.333874 20678 raft_consensus.cc:504] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23c7ec1cf3346698035f0e137e5f578" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39165 } }
I20260620 06:11:51.334237 20678 leader_election.cc:302] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b23c7ec1cf3346698035f0e137e5f578; no voters:
I20260620 06:11:51.334590 20678 leader_election.cc:288] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:51.334712 20683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:51.334950 20683 raft_consensus.cc:686] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [term 1 LEADER]: Becoming Leader. State: Replica: b23c7ec1cf3346698035f0e137e5f578, State: Running, Role: LEADER
I20260620 06:11:51.335273 20683 consensus_queue.cc:237] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23c7ec1cf3346698035f0e137e5f578" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39165 } }
I20260620 06:11:51.335923 20678 sys_catalog.cc:564] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:51.336285 20683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b23c7ec1cf3346698035f0e137e5f578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23c7ec1cf3346698035f0e137e5f578" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39165 } } }
I20260620 06:11:51.336413 20683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:51.336813 20687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:51.336864 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b23c7ec1cf3346698035f0e137e5f578. Latest consensus state: current_term: 1 leader_uuid: "b23c7ec1cf3346698035f0e137e5f578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23c7ec1cf3346698035f0e137e5f578" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39165 } } }
I20260620 06:11:51.336967 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:51.339793 20687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:51.345373 20687 catalog_manager.cc:1347] Generated new cluster ID: 24b2cb872be94adda8ab88558f6eb240
I20260620 06:11:51.345455 20687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:51.356243 20687 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:51.357069 20687 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:51.369107 20687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b23c7ec1cf3346698035f0e137e5f578: Generated new TSK 0
I20260620 06:11:51.369805 20687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:51.424615 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:51.388247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.14.209.190:39165)
0620 06:11:51.388385 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:51.388390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:51.388570 (+ 180us) server_negotiation.cc:407] Connection header received
0620 06:11:51.388726 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:51.388728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:51.388771 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:51.388854 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:11:51.389117 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.389623 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.390833 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.391036 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.391173 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:51.391486 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:11:51.424259 (+ 32773us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:11:51.424492 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 06:11:51.424705 20681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 06:11:51.424890 20707 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:51.388174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:39165 (local address 127.0.0.1:38222)
0620 06:11:51.388474 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:51.388490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:51.388609 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:51.388880 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:51.388883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:51.388891 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:11:51.389004 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:51.389010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.389730 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.389734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:51.390711 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:51.390721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:51.391195 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:51.391199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:51.391295 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:51.391322 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:11:51.424499 (+ 33177us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:11:51.424705 (+ 206us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.209.190:39165: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":119,"threads_started":1}
I20260620 06:11:51.425465 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20611
2026-06-20T06:11:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (313 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:51 dist-test-slave-zt01 krb5kdc[20719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:51 dist-test-slave-zt01 krb5kdc[20719](info): set up 2 sockets
Jun 20 06:11:51 dist-test-slave-zt01 krb5kdc[20719](info): commencing operation
krb5kdc: starting...
W20260620 06:11:52.687289 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:52 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:11:52.805629 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45863
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:52.909179 20732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:52.909430 20732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:52.909500 20732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:52.913355 20732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:52.913434 20732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:52.913471 20732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:52.913528 20732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:52.913563 20732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:52.918496 20732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:52.919413 20732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:52.920341 20732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:52.928170 20740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:52.928189 20737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:52.928189 20738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:52.928223 20732 server_base.cc:1034] running on GCE node
I20260620 06:11:52.928844 20732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:52.929778 20732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:52.930935 20732 hybrid_clock.cc:648] HybridClock initialized: now 1781935912930902 us; error 45 us; skew 500 ppm
Jun 20 06:11:52 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:52.933501 20732 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:11:52.934290 20732 webserver.cc:460] Webserver started at http://127.14.209.190:37819/ using document root <none> and password file <none>
I20260620 06:11:52.934787 20732 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:52.934852 20732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:52.935124 20732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:52.937011 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "5c57bab5c9b54381bac9af53778e9d2b"
format_stamp: "Formatted at 2026-06-20 06:11:52 on dist-test-slave-zt01"
server_key: "f0e69d48b2d67326cce950e35e7e8647"
server_key_iv: "bd7a8e9298934d2b4f404669875f2999"
server_key_version: "encryptionkey@0"
I20260620 06:11:52.937526 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "5c57bab5c9b54381bac9af53778e9d2b"
format_stamp: "Formatted at 2026-06-20 06:11:52 on dist-test-slave-zt01"
server_key: "f0e69d48b2d67326cce950e35e7e8647"
server_key_iv: "bd7a8e9298934d2b4f404669875f2999"
server_key_version: "encryptionkey@0"
I20260620 06:11:52.940827 20732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 06:11:52.942975 20747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:52.944078 20732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:52.944223 20732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "5c57bab5c9b54381bac9af53778e9d2b"
format_stamp: "Formatted at 2026-06-20 06:11:52 on dist-test-slave-zt01"
server_key: "f0e69d48b2d67326cce950e35e7e8647"
server_key_iv: "bd7a8e9298934d2b4f404669875f2999"
server_key_version: "encryptionkey@0"
I20260620 06:11:52.944325 20732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:52.954579 20732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:52.957059 20732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:52.957242 20732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:52.963656 20732 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:45863
I20260620 06:11:52.963685 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:45863 every 8 connection(s)
I20260620 06:11:52.964465 20732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:52.966806 20800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:52.970808 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20732
I20260620 06:11:52.970909 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:52.971307 15174 external_mini_cluster.cc:1428] Setting key daccb76298fc590ce6c37ac97454ac6d
I20260620 06:11:52.971899 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b: Bootstrap starting.
I20260620 06:11:52.974126 20800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:52.974810 20800 log.cc:824] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:52.976531 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b: No bootstrap required, opened a new log
Jun 20 06:11:52 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:52.980126 20800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c57bab5c9b54381bac9af53778e9d2b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45863 } }
I20260620 06:11:52.980406 20800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:52.980487 20800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c57bab5c9b54381bac9af53778e9d2b, State: Initialized, Role: FOLLOWER
I20260620 06:11:52.981005 20800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c57bab5c9b54381bac9af53778e9d2b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45863 } }
I20260620 06:11:52.981139 20800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:52.981228 20800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:52.981324 20800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:52.982103 20800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c57bab5c9b54381bac9af53778e9d2b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45863 } }
I20260620 06:11:52.982414 20800 leader_election.cc:302] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c57bab5c9b54381bac9af53778e9d2b; no voters:
I20260620 06:11:52.982708 20800 leader_election.cc:288] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:52.982805 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:52.983003 20805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [term 1 LEADER]: Becoming Leader. State: Replica: 5c57bab5c9b54381bac9af53778e9d2b, State: Running, Role: LEADER
I20260620 06:11:52.983314 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c57bab5c9b54381bac9af53778e9d2b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45863 } }
I20260620 06:11:52.983981 20800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:52.984416 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c57bab5c9b54381bac9af53778e9d2b. Latest consensus state: current_term: 1 leader_uuid: "5c57bab5c9b54381bac9af53778e9d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c57bab5c9b54381bac9af53778e9d2b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45863 } } }
I20260620 06:11:52.984519 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:52.984892 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:52.972585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.14.209.190:45863)
0620 06:11:52.972973 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:52.972988 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:52.973023 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:11:52.973645 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:52.973661 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:52.973950 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:52.974254 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:52.975102 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:52.976042 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:52.976871 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:52.977313 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:52.977478 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:52.979847 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:52.979852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:52.979863 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:52.979886 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:52.981780 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:52.982199 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:52.982205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:52.982211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:52.982337 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:52.982622 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:52.982625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:52.982627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:52.983171 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:52.983394 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:52.983694 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 06:11:52.983957 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260620 06:11:52.985095 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c57bab5c9b54381bac9af53778e9d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c57bab5c9b54381bac9af53778e9d2b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45863 } } }
I20260620 06:11:52.985180 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:52.988301 20820 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:52.988389 20820 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:52.988463 20809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:52.989131 20809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:52.993654 20809 catalog_manager.cc:1347] Generated new cluster ID: 18f5622ff6624f3096268824a693c940
I20260620 06:11:52.993713 20809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:53.014647 20809 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:53.015708 20809 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:53.026543 20809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c57bab5c9b54381bac9af53778e9d2b: Generated new TSK 0
I20260620 06:11:53.027226 20809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:11:53.061455 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45863
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:53.175231 20826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:53.175446 20826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:53.175485 20826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:53.179443 20826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:53.179493 20826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:53.179549 20826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:53.184667 20826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:53.185642 20826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:53.186602 20826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:53.195389 20832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:53.195483 20831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:53.195488 20834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:53.195745 20826 server_base.cc:1034] running on GCE node
I20260620 06:11:53.196099 20826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:53.196709 20826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:53.197904 20826 hybrid_clock.cc:648] HybridClock initialized: now 1781935913197888 us; error 28 us; skew 500 ppm
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.206988 20826 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:11:53.207840 20826 webserver.cc:460] Webserver started at http://127.14.209.129:33121/ using document root <none> and password file <none>
I20260620 06:11:53.208320 20826 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:53.208375 20826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:53.208587 20826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:53.210467 20826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "ca195ee463e84268a8aad09127ee39a9"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "a765e12efa7e5d035bb269657095a945"
server_key_iv: "dbb2234a207cebdad7057c48834407fa"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.211011 20826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "ca195ee463e84268a8aad09127ee39a9"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "a765e12efa7e5d035bb269657095a945"
server_key_iv: "dbb2234a207cebdad7057c48834407fa"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.214399 20826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:11:53.216792 20841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.217944 20826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 06:11:53.218065 20826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "ca195ee463e84268a8aad09127ee39a9"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "a765e12efa7e5d035bb269657095a945"
server_key_iv: "dbb2234a207cebdad7057c48834407fa"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.218163 20826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:53.237803 20826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:53.240679 20826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:53.240862 20826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:53.241390 20826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:53.242141 20826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:53.242185 20826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.242246 20826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:53.242277 20826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.251168 20826 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:44317
I20260620 06:11:53.251194 20954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:44317 every 8 connection(s)
I20260620 06:11:53.251945 20826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:53.257230 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20826
I20260620 06:11:53.257395 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:53.257647 15174 external_mini_cluster.cc:1428] Setting key 8d4fcb04d05477297198434f5abf836f
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:53.265127 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.253686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:54197 (local address 127.14.209.190:45863)
0620 06:11:53.253854 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.253860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.254679 (+ 819us) server_negotiation.cc:407] Connection header received
0620 06:11:53.255992 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.255995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.256040 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.256118 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.257145 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.257784 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.258861 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.259176 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.259288 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.261654 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:53.261660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:53.261667 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:53.261689 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:53.263225 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.263709 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.263712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.263714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.263793 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.264121 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.264124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.264126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.264295 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:53.264401 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:53.264834 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 06:11:53.264962 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:11:53.265856 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.254097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45863 (local address 127.14.209.129:54197)
0620 06:11:53.254529 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:53.254568 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:53.255792 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:53.256262 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:53.256268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:53.256593 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:53.257005 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:53.257016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.257899 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.257902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:53.258736 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:53.258744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.259314 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.259318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:53.259480 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.260087 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:53.260101 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:53.261496 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:53.263333 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:53.263339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:53.263351 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:53.263600 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:53.263882 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:53.263886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:53.263888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:53.264015 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:53.264397 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:53.264403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:53.264621 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 06:11:53.264796 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 06:11:53.265085 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:11:53.266944 20955 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45863
I20260620 06:11:53.267212 20955 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:53.267637 20955 heartbeater.cc:511] Master 127.14.209.190:45863 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:11:53.269177 20764 ts_manager.cc:194] Registered new tserver with Master: ca195ee463e84268a8aad09127ee39a9 (127.14.209.129:44317)
I20260620 06:11:53.270236 20764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:54197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:11:53.282464 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45863
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:53.387912 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:53.388132 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:53.388171 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:53.392081 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:53.392124 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:53.392211 20960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:53.397215 20960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:53.398092 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:53.399075 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:53.407939 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:53.407974 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:53.407953 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:53.408206 20960 server_base.cc:1034] running on GCE node
I20260620 06:11:53.408607 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:53.409230 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:53.410414 20960 hybrid_clock.cc:648] HybridClock initialized: now 1781935913410387 us; error 29 us; skew 500 ppm
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.413277 20960 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:11:53.414076 20960 webserver.cc:460] Webserver started at http://127.14.209.130:35127/ using document root <none> and password file <none>
I20260620 06:11:53.414543 20960 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:53.414605 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:53.414750 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:53.416601 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "b587cb4df1a04587935d7fdd70644475"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "0d00b7df9de8b2c9a6ba836e2453906e"
server_key_iv: "7e887eddef710925676388da31bc49ba"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.417105 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "b587cb4df1a04587935d7fdd70644475"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "0d00b7df9de8b2c9a6ba836e2453906e"
server_key_iv: "7e887eddef710925676388da31bc49ba"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.420478 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:53.422816 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.424109 20960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:53.424237 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "b587cb4df1a04587935d7fdd70644475"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "0d00b7df9de8b2c9a6ba836e2453906e"
server_key_iv: "7e887eddef710925676388da31bc49ba"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.424341 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:53.442865 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:53.445928 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:53.446143 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:53.446738 20960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:53.447717 20960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:53.447794 20960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.447868 20960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:53.447906 20960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.456969 20960 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:38417
I20260620 06:11:53.456991 21088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:38417 every 8 connection(s)
I20260620 06:11:53.457821 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:53.458228 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 20960
I20260620 06:11:53.458361 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:53.458730 15174 external_mini_cluster.cc:1428] Setting key 272a9df5b7c298e38c90a9440e79ba44
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:11:53.473191 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.459800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:40259 (local address 127.14.209.190:45863)
0620 06:11:53.459914 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.459918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.460897 (+ 979us) server_negotiation.cc:407] Connection header received
0620 06:11:53.462814 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.462816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.462859 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.462921 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.464086 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.464593 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.466545 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.466781 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.466864 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.469707 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:53.469717 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:53.469720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:53.469742 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:53.471227 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.471714 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.471719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.471721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.471800 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.472122 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.472127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.472129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.472300 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:53.472405 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:53.472902 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 06:11:53.473028 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:11:53.473956 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.460267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45863 (local address 127.14.209.130:40259)
0620 06:11:53.460734 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:53.460785 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:53.462561 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:53.463137 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:53.463152 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:53.463495 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:53.463951 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:53.463964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.464724 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.464731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:53.466399 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:53.466416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.466956 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.466963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:53.467249 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.467879 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:53.467903 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:53.469550 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:53.471348 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:53.471353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:53.471366 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:53.471612 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:53.471891 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:53.471895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:53.471897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:53.472027 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:53.472403 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:53.472409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:53.472686 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 06:11:53.472862 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 06:11:53.473140 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":138,"threads_started":1}
I20260620 06:11:53.475262 21089 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45863
I20260620 06:11:53.475478 21089 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:53.475976 21089 heartbeater.cc:511] Master 127.14.209.190:45863 requested a full tablet report, sending...
I20260620 06:11:53.476917 20764 ts_manager.cc:194] Registered new tserver with Master: b587cb4df1a04587935d7fdd70644475 (127.14.209.130:38417)
I20260620 06:11:53.477463 20764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:40259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:11:53.487975 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45863
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:53.591930 21094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:53.592202 21094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:53.592278 21094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:53.596148 21094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:53.596223 21094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:53.596321 21094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:53.601511 21094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:53.602375 21094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:53.603310 21094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:53.612246 21099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:53.612294 21100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:53.612444 21094 server_base.cc:1034] running on GCE node
W20260620 06:11:53.612306 21102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:53.612917 21094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:53.613463 21094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:53.614630 21094 hybrid_clock.cc:648] HybridClock initialized: now 1781935913614606 us; error 36 us; skew 500 ppm
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.617225 21094 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:11:53.618022 21094 webserver.cc:460] Webserver started at http://127.14.209.131:45347/ using document root <none> and password file <none>
I20260620 06:11:53.618520 21094 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:53.618602 21094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:53.618808 21094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:53.620642 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "979e0115efec4c5e8111a7c0d6773a33"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "2e29b3b45a63d8b939476c0bac677f9d"
server_key_iv: "ab3e6a80a4dc13937e8c7a27d711cac2"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.621165 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "979e0115efec4c5e8111a7c0d6773a33"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "2e29b3b45a63d8b939476c0bac677f9d"
server_key_iv: "ab3e6a80a4dc13937e8c7a27d711cac2"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.624425 21094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 06:11:53.626533 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.627681 21094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:53.627799 21094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "979e0115efec4c5e8111a7c0d6773a33"
format_stamp: "Formatted at 2026-06-20 06:11:53 on dist-test-slave-zt01"
server_key: "2e29b3b45a63d8b939476c0bac677f9d"
server_key_iv: "ab3e6a80a4dc13937e8c7a27d711cac2"
server_key_version: "encryptionkey@0"
I20260620 06:11:53.627942 21094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:53.653627 21094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:53.656397 21094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:53.656590 21094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:53.657135 21094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:53.657866 21094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:53.657940 21094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.658005 21094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:53.658061 21094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:53.667186 21094 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:37605
I20260620 06:11:53.667240 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:37605 every 8 connection(s)
I20260620 06:11:53.668090 21094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:53.673277 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21094
I20260620 06:11:53.673424 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:53.673787 15174 external_mini_cluster.cc:1428] Setting key 0403999e7049f293136d4621864d55b7
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:11:53.683936 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.670601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:43213 (local address 127.14.209.190:45863)
0620 06:11:53.670768 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.670775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.671403 (+ 628us) server_negotiation.cc:407] Connection header received
0620 06:11:53.672875 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.672880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.672944 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.673063 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.674790 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.675799 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.677328 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.677672 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.677784 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.680862 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:53.680865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:53.680867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:53.680890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:53.682147 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.682626 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.682629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.682630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.682704 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.683007 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.683009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.683010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.683235 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:53.683309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:53.683733 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 06:11:53.683825 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:11:53.684701 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.670669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45863 (local address 127.14.209.131:43213)
0620 06:11:53.671230 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:53.671282 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:53.672627 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:53.673368 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:53.673379 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:53.673879 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:53.674606 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:53.674624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.675948 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.675954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:53.677217 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:53.677227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.677978 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.677982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:53.678180 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.678855 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:53.678871 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:53.680698 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:53.682260 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:53.682265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:53.682275 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:53.682537 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:53.682783 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:53.682785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:53.682786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:53.682909 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:53.683316 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:53.683320 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:53.683539 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 06:11:53.683687 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 06:11:53.683912 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":180,"threads_started":1}
I20260620 06:11:53.685791 21223 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45863
I20260620 06:11:53.686034 21223 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:53.686503 21223 heartbeater.cc:511] Master 127.14.209.190:45863 requested a full tablet report, sending...
I20260620 06:11:53.687585 20764 ts_manager.cc:194] Registered new tserver with Master: 979e0115efec4c5e8111a7c0d6773a33 (127.14.209.131:37605)
I20260620 06:11:53.688205 20764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:43213
I20260620 06:11:53.689672 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/test-admin.keytab.
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.704753 15174 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 06:11:53.707366 21230 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:53.706031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.14.209.129:44317)
0620 06:11:53.706328 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.706337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.706356 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:11:53.706400 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.706405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.706558 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.706671 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.707171 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51752: BlockingRecv error: recv got EOF from 127.0.0.1:51752 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":120,"threads_started":1}
W20260620 06:11:53.707619 15174 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.708523 15174 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.709367 15174 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:53.718681 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.710106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.14.209.129:44317)
0620 06:11:53.710258 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.710261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.710275 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:11:53.710337 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.710339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.710373 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.710454 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.710943 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.711812 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.712704 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.713108 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.713197 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.715511 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:53.715519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:53.715529 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:53.715551 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:53.717152 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.717550 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.717554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.717556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.717629 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.717855 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.717857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.717858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.718080 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:53.718220 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:53.718386 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 06:11:53.718559 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 06:11:53.722138 21230 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:53.721267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.14.209.129:44317)
0620 06:11:53.721371 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.721374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.721406 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:11:53.721511 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.721513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.721543 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.721606 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.722036 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51764: BlockingRecv error: recv got EOF from 127.0.0.1:51764 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 06:11:53.722189 15174 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.723348 15174 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.724293 15174 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:53.732596 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.725056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.14.209.129:44317)
0620 06:11:53.725164 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.725168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.725252 (+ 84us) server_negotiation.cc:407] Connection header received
0620 06:11:53.725380 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.725383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.725421 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.725512 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.725969 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.726453 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.727283 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.727509 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.727617 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.729680 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:53.729684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:53.729689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:53.729713 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:53.731189 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.731569 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.731574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.731576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.731651 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.731906 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.731910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.731912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.732076 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:53.732211 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:53.732422 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 06:11:53.732509 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 06:11:53.734833 21230 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:53.734020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.14.209.129:44317)
0620 06:11:53.734114 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.734118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.734178 (+ 60us) server_negotiation.cc:407] Connection header received
0620 06:11:53.734283 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.734288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.734320 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.734405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.734758 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51768: BlockingRecv error: recv got EOF from 127.0.0.1:51768 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 06:11:53.735270 15174 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935885709390-15174-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.736193 15174 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:53.737018 15174 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:11:53.744508 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:53.737701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.14.209.129:44317)
0620 06:11:53.737823 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:53.737826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:53.737838 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:53.737949 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:53.737951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:53.737996 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:53.738058 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:53.738465 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.738898 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.739657 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:53.739852 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:53.739926 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:53.741868 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:53.741871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:53.741873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:53.741892 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:53.743046 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.743462 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.743465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.743466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.743539 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:53.743815 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:53.743817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:53.743818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:53.743948 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:53.744068 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:53.744242 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 06:11:53.744356 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:11:53.745325 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20826
I20260620 06:11:53.751188 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20960
I20260620 06:11:53.758507 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21094
I20260620 06:11:53.764263 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 20732
2026-06-20T06:11:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2105 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[21243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[21243](info): set up 2 sockets
Jun 20 06:11:53 dist-test-slave-zt01 krb5kdc[21243](info): commencing operation
krb5kdc: starting...
W20260620 06:11:54.831457 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:11:54 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab.
Entry for principal oryx/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab.
I20260620 06:11:54.949921 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46813
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab
--principal=oryx/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:55.054893 21256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:55.055174 21256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:55.055219 21256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:55.059032 21256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:55.059093 21256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:55.059113 21256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:55.059128 21256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:55.059181 21256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:55.064041 21256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab
--principal=oryx/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:55.064908 21256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:55.065795 21256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:55.073992 21262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.074117 21256 server_base.cc:1034] running on GCE node
W20260620 06:11:55.073961 21261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.073951 21264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.074815 21256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:55.075821 21256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:55.077006 21256 hybrid_clock.cc:648] HybridClock initialized: now 1781935915076979 us; error 40 us; skew 500 ppm
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:55.079876 21256 init.cc:377] Logged in from keytab as oryx/127.14.209.190@KRBTEST.COM (short username oryx)
I20260620 06:11:55.080732 21256 webserver.cc:460] Webserver started at http://127.14.209.190:35529/ using document root <none> and password file <none>
I20260620 06:11:55.081265 21256 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:55.081322 21256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:55.081518 21256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:55.083433 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "9f5583e026484a63a4eaeaf5dc5afc4c"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "1c254f5b1feb5a3d092582ccfb6807d0"
server_key_iv: "4175ff156b5edaaf7c7d6201632a6abc"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.083956 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "9f5583e026484a63a4eaeaf5dc5afc4c"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "1c254f5b1feb5a3d092582ccfb6807d0"
server_key_iv: "4175ff156b5edaaf7c7d6201632a6abc"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.087426 21256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:55.089627 21271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.090767 21256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:55.090883 21256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "9f5583e026484a63a4eaeaf5dc5afc4c"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "1c254f5b1feb5a3d092582ccfb6807d0"
server_key_iv: "4175ff156b5edaaf7c7d6201632a6abc"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.090979 21256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:55.116459 21256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:55.119515 21256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:55.119701 21256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:55.126566 21256 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:46813
I20260620 06:11:55.126856 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:46813 every 8 connection(s)
I20260620 06:11:55.127527 21256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:55.130043 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:55.135241 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Bootstrap starting.
I20260620 06:11:55.135444 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21256
I20260620 06:11:55.135577 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:11:55.135848 15174 external_mini_cluster.cc:1428] Setting key 360f657135c17017230fa8e6d1422dfa
I20260620 06:11:55.137743 21324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:55.138420 21324 log.cc:824] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:55.140417 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: No bootstrap required, opened a new log
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:11:55.144843 21324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } }
I20260620 06:11:55.145102 21324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:55.145180 21324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5583e026484a63a4eaeaf5dc5afc4c, State: Initialized, Role: FOLLOWER
I20260620 06:11:55.145673 21324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } }
I20260620 06:11:55.145838 21324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:55.145901 21324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:55.146004 21324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:55.147192 21324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } }
I20260620 06:11:55.147567 21324 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f5583e026484a63a4eaeaf5dc5afc4c; no voters:
I20260620 06:11:55.147889 21324 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:55.148010 21329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:55.148237 21329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 LEADER]: Becoming Leader. State: Replica: 9f5583e026484a63a4eaeaf5dc5afc4c, State: Running, Role: LEADER
I20260620 06:11:55.148604 21329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } }
I20260620 06:11:55.149060 21324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:55.150179 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5583e026484a63a4eaeaf5dc5afc4c. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } }
I20260620 06:11:55.150290 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:55.150607 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } }
I20260620 06:11:55.150679 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:55.151731 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.137120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.14.209.190:46813)
0620 06:11:55.137508 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:55.137533 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:55.137566 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:55.138218 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:55.138236 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:55.138564 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:55.138909 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:55.139765 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.140786 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.141893 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.142361 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.142480 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.145428 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:55.145434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:55.145450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:55.145480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:55.147552 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.148123 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.148129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.148135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.148281 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.148606 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.148609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.148612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.149073 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:55.149259 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:55.150121 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 06:11:55.150426 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
I20260620 06:11:55.151265 21338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:55.154810 21338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:55.159935 21338 catalog_manager.cc:1347] Generated new cluster ID: e373f05ea2c3438e82bb64d3fba9d0ad
I20260620 06:11:55.159991 21338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:55.166951 21338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:55.168120 21338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:55.175014 21338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Generated new TSK 0
I20260620 06:11:55.175625 21338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab.
Entry for principal oryx/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab.
I20260620 06:11:55.206872 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46813
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:55.314334 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:55.314616 21350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:55.314692 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:55.318831 21350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:55.318939 21350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:55.319041 21350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:55.324101 21350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:55.325068 21350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:55.326033 21350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:55.334940 21358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.334971 21356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.335179 21350 server_base.cc:1034] running on GCE node
W20260620 06:11:55.335263 21355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.335671 21350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:55.336244 21350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:55.337423 21350 hybrid_clock.cc:648] HybridClock initialized: now 1781935915337404 us; error 32 us; skew 500 ppm
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:55.340157 21350 init.cc:377] Logged in from keytab as oryx/127.14.209.129@KRBTEST.COM (short username oryx)
I20260620 06:11:55.340922 21350 webserver.cc:460] Webserver started at http://127.14.209.129:33761/ using document root <none> and password file <none>
I20260620 06:11:55.341431 21350 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:55.341495 21350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:55.341728 21350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:55.343636 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "49d206ba20aa4807b8070b2ef009df51"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "802d1273d289509a3bd10ca19251ada0"
server_key_iv: "ef22dfa62bdf699c0cbb17c4c82585a9"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.344162 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "49d206ba20aa4807b8070b2ef009df51"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "802d1273d289509a3bd10ca19251ada0"
server_key_iv: "ef22dfa62bdf699c0cbb17c4c82585a9"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.347570 21350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:55.349872 21365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.351027 21350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:55.351195 21350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "49d206ba20aa4807b8070b2ef009df51"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "802d1273d289509a3bd10ca19251ada0"
server_key_iv: "ef22dfa62bdf699c0cbb17c4c82585a9"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.351286 21350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:55.368310 21350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:55.371423 21350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:55.371619 21350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:55.372190 21350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:55.372932 21350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:55.372977 21350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.373044 21350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:55.373075 21350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.381407 21350 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:39157
I20260620 06:11:55.381443 21478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:39157 every 8 connection(s)
I20260620 06:11:55.382192 21350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:55.382699 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21350
I20260620 06:11:55.382845 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:11:55.383195 15174 external_mini_cluster.cc:1428] Setting key aa073859f8a37ab011fb268bb87b878a
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.209.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.130@KRBTEST.COM" created.
I20260620 06:11:55.396081 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.383972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:47931 (local address 127.14.209.190:46813)
0620 06:11:55.384101 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:55.384104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:55.385023 (+ 919us) server_negotiation.cc:407] Connection header received
0620 06:11:55.386643 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:55.386645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:55.386689 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:55.386760 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:55.387954 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.388503 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.389600 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.389836 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.389945 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.392495 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:55.392504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:55.392506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:55.392527 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:55.394012 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.394514 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.394520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.394521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.394601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.394965 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.394969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.394970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.395225 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:55.395329 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:55.395813 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 06:11:55.395924 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:11:55.396788 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.384411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:47931)
0620 06:11:55.384864 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:55.384913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:55.386390 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:55.386940 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:55.386951 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:55.387400 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:55.387831 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:55.387843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.388627 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.388631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:55.389476 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:55.389485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.389978 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.389981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:55.390144 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.390690 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:55.390706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:55.392333 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:55.394143 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:55.394148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:55.394160 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:55.394404 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:55.394706 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:55.394709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:55.394711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:55.394847 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:55.395337 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:55.395345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:55.395561 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 06:11:55.395772 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:11:55.396071 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":134,"threads_started":1}
I20260620 06:11:55.397989 21479 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46813
I20260620 06:11:55.398226 21479 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:55.398681 21479 heartbeater.cc:511] Master 127.14.209.190:46813 requested a full tablet report, sending...
I20260620 06:11:55.400178 21288 ts_manager.cc:194] Registered new tserver with Master: 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129:39157)
I20260620 06:11:55.401242 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:47931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab.
Entry for principal oryx/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab.
I20260620 06:11:55.413357 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46813
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:55.519157 21484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:55.519371 21484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:55.519418 21484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:55.523280 21484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:55.523339 21484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:55.523427 21484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:55.528383 21484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:55.529249 21484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:55.530143 21484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:55.539515 21490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.539620 21492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.539523 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.539763 21484 server_base.cc:1034] running on GCE node
I20260620 06:11:55.540303 21484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:55.540906 21484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:55.542078 21484 hybrid_clock.cc:648] HybridClock initialized: now 1781935915542051 us; error 42 us; skew 500 ppm
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:55.544852 21484 init.cc:377] Logged in from keytab as oryx/127.14.209.130@KRBTEST.COM (short username oryx)
I20260620 06:11:55.545799 21484 webserver.cc:460] Webserver started at http://127.14.209.130:46741/ using document root <none> and password file <none>
I20260620 06:11:55.546348 21484 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:55.546439 21484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:55.546664 21484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:55.548643 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "c079e9d31932b6f30cabbcc39a6115d7"
server_key_iv: "a17bf6e1bfcf65332fb39d1600d503b1"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.549254 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "c079e9d31932b6f30cabbcc39a6115d7"
server_key_iv: "a17bf6e1bfcf65332fb39d1600d503b1"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.552816 21484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 06:11:55.555111 21499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.556351 21484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:55.556445 21484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "c079e9d31932b6f30cabbcc39a6115d7"
server_key_iv: "a17bf6e1bfcf65332fb39d1600d503b1"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.556581 21484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:55.583412 21484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:55.586114 21484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:55.586302 21484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:55.586855 21484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:55.587651 21484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:55.587697 21484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.587765 21484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:55.587807 21484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.596381 21484 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:42033
I20260620 06:11:55.596421 21612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:42033 every 8 connection(s)
I20260620 06:11:55.597210 21484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:55.599372 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21484
I20260620 06:11:55.599491 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:11:55.599836 15174 external_mini_cluster.cc:1428] Setting key ea53c3f933189cd9268196e9b04b3ffd
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.131@KRBTEST.COM" created.
I20260620 06:11:55.611315 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.598861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:35437 (local address 127.14.209.190:46813)
0620 06:11:55.599043 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:55.599094 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:55.600115 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 06:11:55.601642 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:55.601645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:55.601696 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:55.601772 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:55.602910 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.603808 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.604998 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.605231 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.605340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.607672 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:55.607676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:55.607678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:55.607702 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:55.609293 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.609827 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.609832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.609834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.609916 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.610265 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.610268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.610269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.610454 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:55.610562 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:55.611014 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 06:11:55.611151 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 06:11:55.612211 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.599418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:35437)
0620 06:11:55.599932 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:55.600020 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:55.601442 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:55.601929 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:55.601935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:55.602265 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:55.602741 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:55.602764 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.603950 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.603962 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:55.604833 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:55.604842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.605372 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.605376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:55.605559 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.606115 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:55.606130 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:55.607517 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:55.609446 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:55.609451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:55.609464 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:55.609720 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:55.610018 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:55.610021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:55.610023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:55.610158 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:55.610561 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:55.610567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:55.610778 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 06:11:55.610972 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 06:11:55.611282 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":145,"threads_started":1}
I20260620 06:11:55.613358 21613 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46813
I20260620 06:11:55.613603 21613 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:55.614039 21613 heartbeater.cc:511] Master 127.14.209.190:46813 requested a full tablet report, sending...
I20260620 06:11:55.615163 21288 ts_manager.cc:194] Registered new tserver with Master: c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033)
I20260620 06:11:55.615707 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:35437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab.
Entry for principal oryx/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab.
I20260620 06:11:55.627120 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46813
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:55.730404 21618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:55.730624 21618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:55.730664 21618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:55.734680 21618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:55.734731 21618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:55.734818 21618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:55.739811 21618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:55.740684 21618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:55.741586 21618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:55.750537 21626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.750572 21624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.750550 21623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.750919 21618 server_base.cc:1034] running on GCE node
I20260620 06:11:55.751297 21618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:55.751852 21618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:55.753034 21618 hybrid_clock.cc:648] HybridClock initialized: now 1781935915753026 us; error 28 us; skew 500 ppm
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:55.755864 21618 init.cc:377] Logged in from keytab as oryx/127.14.209.131@KRBTEST.COM (short username oryx)
I20260620 06:11:55.756776 21618 webserver.cc:460] Webserver started at http://127.14.209.131:42141/ using document root <none> and password file <none>
I20260620 06:11:55.757396 21618 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:55.757478 21618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:55.757730 21618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:55.760150 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "aee8328a367745b286e8981448426b46"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "165250be7e1dbf170527568913a29e9b"
server_key_iv: "1d0c528e9a9ca2391adf825476477fae"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.760700 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "aee8328a367745b286e8981448426b46"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "165250be7e1dbf170527568913a29e9b"
server_key_iv: "1d0c528e9a9ca2391adf825476477fae"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.764108 21618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 06:11:55.766258 21633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.767452 21618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:11:55.767584 21618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "aee8328a367745b286e8981448426b46"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "165250be7e1dbf170527568913a29e9b"
server_key_iv: "1d0c528e9a9ca2391adf825476477fae"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.767688 21618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:55.779714 21618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:55.782670 21618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:55.782853 21618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:55.783457 21618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:55.784207 21618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:55.784277 21618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.784341 21618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:55.784391 21618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.793170 21618 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:39817
I20260620 06:11:55.793231 21746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:39817 every 8 connection(s)
I20260620 06:11:55.794026 21618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:55.802538 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21618
I20260620 06:11:55.802706 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:11:55.803014 15174 external_mini_cluster.cc:1428] Setting key 3c787a945437953d2f0d7ca33988b4b1
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:11:55.810276 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.795700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:47461 (local address 127.14.209.190:46813)
0620 06:11:55.795831 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:55.795835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:55.796555 (+ 720us) server_negotiation.cc:407] Connection header received
0620 06:11:55.797885 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:55.797889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:55.797950 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:55.798060 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:55.799149 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.799676 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.800797 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.801026 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.801139 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.806690 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:55.806693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:55.806695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:55.806718 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:55.808346 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.808825 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.808827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.808829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.808902 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:55.809259 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:55.809262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:55.809263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:55.809439 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:55.809520 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:55.810053 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 06:11:55.810148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:55.811020 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:55.795982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:47461)
0620 06:11:55.796388 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:55.796429 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:55.797681 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:55.798216 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:55.798223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:55.798550 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:55.798977 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:55.798989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.799801 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.799807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:55.800665 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:55.800674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:55.801175 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:55.801179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:55.801352 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:55.804967 (+ 3615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:55.804998 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:55.806508 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:55.808459 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:55.808464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:55.808474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:55.808718 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:55.809015 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:55.809018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:55.809019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:55.809157 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:55.809552 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:55.809558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:55.809804 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 06:11:55.809964 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 06:11:55.810224 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260620 06:11:55.812121 21747 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46813
I20260620 06:11:55.812374 21747 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:55.813342 21747 heartbeater.cc:511] Master 127.14.209.190:46813 requested a full tablet report, sending...
I20260620 06:11:55.814677 21288 ts_manager.cc:194] Registered new tserver with Master: aee8328a367745b286e8981448426b46 (127.14.209.131:39817)
I20260620 06:11:55.815266 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:47461
I20260620 06:11:55.817414 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.209.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.189.keytab.
Entry for principal oryx/127.14.209.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.189.keytab.
I20260620 06:11:55.842314 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:34149
--webserver_interface=127.14.209.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.189.keytab
--principal=oryx/127.14.209.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46813,127.14.209.189:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:55.949287 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:55.949514 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:55.949553 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:55.953701 21752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:55.953749 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:55.953768 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:55.953785 21752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:55.953840 21752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:55.959015 21752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46813,127.14.209.189:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.189.keytab
--principal=oryx/127.14.209.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.209.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
W20260620 06:11:55.959215 21752 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.209.190:46813,127.14.209.189:34149'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 06:11:55.960016 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:55.960984 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:55.969197 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.969197 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:55.969439 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:55.969537 21752 server_base.cc:1034] running on GCE node
I20260620 06:11:55.970055 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:55.971117 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:55.972295 21752 hybrid_clock.cc:648] HybridClock initialized: now 1781935915972269 us; error 39 us; skew 500 ppm
Jun 20 06:11:55 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:55.975188 21752 init.cc:377] Logged in from keytab as oryx/127.14.209.189@KRBTEST.COM (short username oryx)
I20260620 06:11:55.976011 21752 webserver.cc:460] Webserver started at http://127.14.209.189:32979/ using document root <none> and password file <none>
I20260620 06:11:55.976547 21752 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:55.976606 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:55.976806 21752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:55.978698 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/instance:
uuid: "a9940460356c4d95a1242f7b59d82633"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "8db34a2a82d488b02981cb10678349b2"
server_key_iv: "8f3747d7cd1bce88d441ca0695f4ae8b"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.979264 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal/instance:
uuid: "a9940460356c4d95a1242f7b59d82633"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "8db34a2a82d488b02981cb10678349b2"
server_key_iv: "8f3747d7cd1bce88d441ca0695f4ae8b"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.982681 21752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:55.985010 21767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:55.986110 21752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:11:55.986233 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
uuid: "a9940460356c4d95a1242f7b59d82633"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "8db34a2a82d488b02981cb10678349b2"
server_key_iv: "8f3747d7cd1bce88d441ca0695f4ae8b"
server_key_version: "encryptionkey@0"
I20260620 06:11:55.986335 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:55.998883 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:56.001578 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:56.001772 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:56.008669 21752 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.189:34149
I20260620 06:11:56.008690 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.189:34149 every 8 connection(s)
I20260620 06:11:56.009541 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
I20260620 06:11:56.012063 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:56.017614 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21752
I20260620 06:11:56.019016 21820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.189@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
I20260620 06:11:56.035089 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.021376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.14.209.190:46813)
0620 06:11:56.022130 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.022137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.022154 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:11:56.022201 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.022206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.022260 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.022353 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.023686 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.024456 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.026566 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.026793 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.026908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.030793 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.030797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.030801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.030832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.032650 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.033242 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.033247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.033249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.033336 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.033673 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.033677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.033679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.033966 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.034071 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.034711 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.034869 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:56.036075 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.021328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:38928)
0620 06:11:56.021675 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.021718 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.021834 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.022443 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.022451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.022886 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.023519 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.023534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.024610 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.024613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.026444 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.026456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.026938 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.026941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.027145 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.028005 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.028026 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.030654 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.032759 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.032765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.032773 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.033135 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.033432 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.033437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.033439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.033576 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.034077 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.034085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.034413 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.034644 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.034986 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":20,"thread_start_us":124,"threads_started":1}
I20260620 06:11:56.037743 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.019564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.14.209.189:34149)
0620 06:11:56.020048 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.020070 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.020116 (+ 46us) server_negotiation.cc:407] Connection header received
0620 06:11:56.021135 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.021160 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.021596 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.022608 (+ 1012us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.023036 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.024762 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.026206 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.026821 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.027372 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.030867 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.030875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.030902 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.030929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.033121 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.033663 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.033669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.033671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.033764 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.036406 (+ 2642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.036410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.036412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.036700 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.036930 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.037382 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.037524 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":325,"thread_start_us":132,"threads_started":1}
I20260620 06:11:56.038249 21820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.189@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
I20260620 06:11:56.050479 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.040521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.14.209.189:34149)
0620 06:11:56.040618 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.040622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.040636 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:11:56.040668 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.040670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.040705 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.040800 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.041260 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.042599 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.043660 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.044108 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.044241 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.046845 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.046849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.046850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.046870 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.048497 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.049004 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.049007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.049008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.049080 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.049474 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.049476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.049477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.049643 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.049718 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.050049 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.050331 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":35}
I20260620 06:11:56.050515 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.039026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:42948)
0620 06:11:56.040367 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.040382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.040458 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.040851 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.040856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.041027 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.041146 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.041153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.042717 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.042721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.043508 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.043518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.044286 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.044292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.044426 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.044910 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.044923 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.046703 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.048608 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.048614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.048618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.048890 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.049269 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.049273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.049276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.049384 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.049759 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.049764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.049875 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.050000 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.050210 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1273}
I20260620 06:11:56.053964 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Bootstrap starting.
I20260620 06:11:56.056948 21820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:56.057682 21820 log.cc:824] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:56.059628 21820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: No bootstrap required, opened a new log
I20260620 06:11:56.062623 21820 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } }
I20260620 06:11:56.062822 21820 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:56.062855 21820 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9940460356c4d95a1242f7b59d82633, State: Initialized, Role: FOLLOWER
I20260620 06:11:56.063330 21820 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } }
I20260620 06:11:56.063764 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } } }
I20260620 06:11:56.063897 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:56.064461 21820 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:11:56.068987 21837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:56.069123 21837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:56.071328 21752 master_runner.cc:428] Detected that this master a9940460356c4d95a1242f7b59d82633 is joining an existing cluster
I20260620 06:11:56.071383 21752 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.209.189:34149
I20260620 06:11:56.081691 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.073210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:42964)
0620 06:11:56.073553 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.073569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.073689 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.073931 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.073934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.074128 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.074226 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.074231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.075159 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.075163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.076261 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.076271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.076895 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.076897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.076990 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.077582 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.077593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.077897 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.080141 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.080152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.080391 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.080691 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.080696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.080785 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.081155 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.081158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.081236 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.081346 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.081485 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":35,"thread_start_us":115,"threads_started":1}
I20260620 06:11:56.081923 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.073435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:38944)
0620 06:11:56.073706 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.073720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.073848 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.074205 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.074207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.074430 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.074564 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.074571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.075549 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.075560 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.076206 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.076215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.076792 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.076796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.076913 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.077821 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.077836 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.078245 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.080207 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.080212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.080485 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.080732 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.080740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.080818 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.081261 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.081265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.081353 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.081484 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.081617 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":25,"thread_start_us":80,"threads_started":1}
I20260620 06:11:56.082075 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.073217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.14.209.189:34149)
0620 06:11:56.073297 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.073301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.073647 (+ 346us) server_negotiation.cc:407] Connection header received
0620 06:11:56.073786 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.073791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.073826 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.073919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.074329 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.074981 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.076411 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.076719 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.076800 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.078049 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.078052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.078055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.078076 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.080023 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080510 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.080513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.080515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.080591 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080880 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.080885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.080888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.081063 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.081166 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.081399 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.081537 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":23}
I20260620 06:11:56.082255 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.073214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.14.209.190:46813)
0620 06:11:56.073326 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.073330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.073789 (+ 459us) server_negotiation.cc:407] Connection header received
0620 06:11:56.073973 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.073976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.074022 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.074073 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.074686 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.075384 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.076335 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.076673 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.076786 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.078386 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.078390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.078395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.078424 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.080109 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080572 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.080575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.080577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.080658 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.080917 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.080920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.080922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.081125 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.081215 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.081713 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.081845 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.190@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
I20260620 06:11:56.095276 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.086204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.14.209.189:34149)
0620 06:11:56.086319 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.086323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.086646 (+ 323us) server_negotiation.cc:407] Connection header received
0620 06:11:56.086816 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.086819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.086867 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.086953 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.087533 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.088278 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.089098 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.089458 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.089568 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.091863 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.091866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.091869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.091896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.093414 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.093941 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.093946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.093948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.094044 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.094345 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.094347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.094349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.094573 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.094651 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.095020 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.095154 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:11:56.095317 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.086140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:42974)
0620 06:11:56.086522 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.086567 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.086690 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.086974 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.086978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.087331 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.087427 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.087433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.088380 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.088382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.088991 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.089000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.089565 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.089567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.089649 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.090120 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.090131 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.091735 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.093532 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.093541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.093546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.093842 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.094149 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.094152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.094154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.094264 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.094676 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.094682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.094797 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.094965 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.095172 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
I20260620 06:11:56.096859 21288 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.209.189:34149: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a9940460356c4d95a1242f7b59d82633"
member_type: NON_VOTER
last_known_addr {
host: "127.14.209.189"
port: 34149
}
attrs {
promote: true
}
}
dest_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c"
cas_config_opid_index: -1
I20260620 06:11:56.097440 21288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } }
I20260620 06:11:56.098901 21848 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a9940460356c4d95a1242f7b59d82633 (127.14.209.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } }
I20260620 06:11:56.099663 21794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.14.209.188@KRBTEST.COM; defaulting to no policy
I20260620 06:11:56.101112 21848 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.209.189:34149
I20260620 06:11:56.101338 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } }
I20260620 06:11:56.101444 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:56.101850 21752 master.cc:537] Master@127.14.209.189:34149 shutting down...
I20260620 06:11:56.102943 21794 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Refusing update from remote peer 9f5583e026484a63a4eaeaf5dc5afc4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 06:11:56.104225 21848 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.188@KRBTEST.COM" created.
W20260620 06:11:56.107925 21275 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 06:11:56.108743 21275 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c -> Peer a9940460356c4d95a1242f7b59d82633 (127.14.209.189:34149): Couldn't send request to peer a9940460356c4d95a1242f7b59d82633. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:11:56.118274 21752 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:56.118570 21752 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:56.118692 21752 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: stopping tablet replica
I20260620 06:11:56.123899 21752 master.cc:552] Master@127.14.209.189:34149 shutdown complete.
I20260620 06:11:56.124006 21752 master_runner.cc:305] Clearing existing system tablet
I20260620 06:11:56.124267 21752 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.124331 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.125439 21752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.188.keytab.
Entry for principal oryx/127.14.209.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.188.keytab.
I20260620 06:11:56.126796 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:43821
--webserver_interface=127.14.209.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.188.keytab
--principal=oryx/127.14.209.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:11:56.131382 21853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.131652 21752 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.131939 21752 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 06:11:56.132031 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
uuid: "a9940460356c4d95a1242f7b59d82633"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "8db34a2a82d488b02981cb10678349b2"
server_key_iv: "8f3747d7cd1bce88d441ca0695f4ae8b"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.134572 21752 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:56.137972 21752 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:11:56.138175 21752 log.cc:1196] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 06:11:56.138514 21752 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Deleting consensus metadata
I20260620 06:11:56.138716 21752 master_runner.cc:315] Copying system tablet from 127.14.209.190:46813
I20260620 06:11:56.139914 21752 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: tablet copy: Beginning tablet copy session from remote peer at address 127.14.209.190:46813
I20260620 06:11:56.149324 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.140610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.14.209.190:46813)
0620 06:11:56.140752 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.140757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.141251 (+ 494us) server_negotiation.cc:407] Connection header received
0620 06:11:56.141428 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.141431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.141487 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.141537 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.142177 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.143012 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.144236 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.144580 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.144693 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.145980 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.145984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.145987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.146012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.147678 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.148132 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.148136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.148138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.148217 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.148503 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.148506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.148508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.148707 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.148796 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.149067 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.149177 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:11:56.149358 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.140840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:38950)
0620 06:11:56.141166 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.141180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.141322 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.141670 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.141672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.141914 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.142046 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.142055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.143168 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.143171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.144112 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.144121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.144722 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.144728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.144831 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.145402 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.145416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.145835 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.147786 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.147790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.147792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.148034 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.148311 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.148314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.148315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.148401 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.148801 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.148804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.148902 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.149007 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.149149 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":87,"threads_started":1}
I20260620 06:11:56.150156 21308 tablet_copy_service.cc:140] P 9f5583e026484a63a4eaeaf5dc5afc4c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a9940460356c4d95a1242f7b59d82633 ({username='oryx', principal='oryx/127.14.209.189@KRBTEST.COM'} at 127.0.0.1:38950)
I20260620 06:11:56.150310 21308 tablet_copy_service.cc:161] P 9f5583e026484a63a4eaeaf5dc5afc4c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a9940460356c4d95a1242f7b59d82633 at {username='oryx', principal='oryx/127.14.209.189@KRBTEST.COM'} at 127.0.0.1:38950: session id = a9940460356c4d95a1242f7b59d82633-00000000000000000000000000000000
I20260620 06:11:56.152132 21308 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:11:56.153597 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:11:56.155900 21752 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: tablet copy: Starting download of 0 data blocks...
I20260620 06:11:56.156075 21752 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: tablet copy: Starting download of 1 WAL segments...
I20260620 06:11:56.157570 21752 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:11:56.159327 21308 tablet_copy_service.cc:342] P 9f5583e026484a63a4eaeaf5dc5afc4c: Request end of tablet copy session a9940460356c4d95a1242f7b59d82633-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.209.189@KRBTEST.COM'} at 127.0.0.1:38950
I20260620 06:11:56.159430 21308 tablet_copy_service.cc:433] P 9f5583e026484a63a4eaeaf5dc5afc4c: ending tablet copy session a9940460356c4d95a1242f7b59d82633-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a9940460356c4d95a1242f7b59d82633
I20260620 06:11:56.166617 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:56.168258 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.168322 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.168536 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.169086 21752 server_base.cc:1034] running on GCE node
I20260620 06:11:56.169261 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:56.169538 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:56.170682 21752 hybrid_clock.cc:648] HybridClock initialized: now 1781935916170669 us; error 27 us; skew 500 ppm
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:56.172335 21752 init.cc:377] Logged in from keytab as oryx/127.14.209.189@KRBTEST.COM (short username oryx)
I20260620 06:11:56.172765 21752 webserver.cc:460] Webserver started at http://127.14.209.189:34583/ using document root <none> and password file <none>
I20260620 06:11:56.172935 21752 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.172982 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.174196 21752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:56.175145 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.175401 21752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:56.175482 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
uuid: "a9940460356c4d95a1242f7b59d82633"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "8db34a2a82d488b02981cb10678349b2"
server_key_iv: "8f3747d7cd1bce88d441ca0695f4ae8b"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.175554 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.212298 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:56.214568 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:56.219305 21752 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.189:34149
I20260620 06:11:56.219403 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.189:34149 every 8 connection(s)
I20260620 06:11:56.219928 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
I20260620 06:11:56.221697 21924 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:11:56.222726 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Bootstrap starting.
I20260620 06:11:56.231920 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:11:56.232195 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Bootstrap complete.
I20260620 06:11:56.232710 21924 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } }
I20260620 06:11:56.232841 21924 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9940460356c4d95a1242f7b59d82633, State: Initialized, Role: LEARNER
I20260620 06:11:56.232944 21924 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } }
I20260620 06:11:56.233253 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } }
I20260620 06:11:56.233357 21924 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:56.233438 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:11:56.238497 21938 catalog_manager.cc:1259] Loaded cluster ID: e373f05ea2c3438e82bb64d3fba9d0ad
I20260620 06:11:56.238612 21938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: loading cluster ID for follower catalog manager: success
I20260620 06:11:56.241044 21938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: acquiring CA information for follower catalog manager: success
I20260620 06:11:56.242547 21938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 06:11:56.259348 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:56.259624 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:56.259696 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:56.263641 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:56.263688 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:56.263707 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:56.263724 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:56.263739 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:56.268648 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.188.keytab
--principal=oryx/127.14.209.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.209.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:56.269474 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:56.270303 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:56.278504 21943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.278543 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.278539 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.278775 21854 server_base.cc:1034] running on GCE node
I20260620 06:11:56.279376 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:56.280368 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:56.281555 21854 hybrid_clock.cc:648] HybridClock initialized: now 1781935916281542 us; error 23 us; skew 500 ppm
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:56.284211 21854 init.cc:377] Logged in from keytab as oryx/127.14.209.188@KRBTEST.COM (short username oryx)
I20260620 06:11:56.284987 21854 webserver.cc:460] Webserver started at http://127.14.209.188:40513/ using document root <none> and password file <none>
I20260620 06:11:56.285502 21854 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.285591 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.285825 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:56.287741 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/instance:
uuid: "138f268bef6d450e85d5b44f59531657"
format_stamp: "Formatted at 2026-06-20 06:11:56 on dist-test-slave-zt01"
server_key: "90893a82bd1ab1e7b3e2626f1a2bb57f"
server_key_iv: "191aab9d08598e27249c3abb107c7da5"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.288271 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal/instance:
uuid: "138f268bef6d450e85d5b44f59531657"
format_stamp: "Formatted at 2026-06-20 06:11:56 on dist-test-slave-zt01"
server_key: "90893a82bd1ab1e7b3e2626f1a2bb57f"
server_key_iv: "191aab9d08598e27249c3abb107c7da5"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.291596 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:56.293741 21953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.294914 21854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:56.295044 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
uuid: "138f268bef6d450e85d5b44f59531657"
format_stamp: "Formatted at 2026-06-20 06:11:56 on dist-test-slave-zt01"
server_key: "90893a82bd1ab1e7b3e2626f1a2bb57f"
server_key_iv: "191aab9d08598e27249c3abb107c7da5"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.295197 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.314459 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:56.317070 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:56.317251 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:56.323784 21854 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.188:43821
I20260620 06:11:56.324059 22005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.188:43821 every 8 connection(s)
I20260620 06:11:56.324667 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
I20260620 06:11:56.327307 22006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:56.330652 22006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:56.332572 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 21854
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.188@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.209.188@KRBTEST.COM
I20260620 06:11:56.343652 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.332428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.14.209.190:46813)
0620 06:11:56.332562 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.332567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.333365 (+ 798us) server_negotiation.cc:407] Connection header received
0620 06:11:56.334537 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.334540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.334591 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.334661 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.335663 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.336221 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.337323 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.337543 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.337650 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.340057 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.340061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.340063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.340085 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.341665 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.342148 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.342152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.342154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.342233 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.342563 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.342568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.342570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.342741 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.342848 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.343370 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.343490 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:11:56.344329 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.333566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.14.209.188:43821)
0620 06:11:56.333890 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.333962 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.334002 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:11:56.334309 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.334329 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.334565 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.334803 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.335640 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.336617 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.337461 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.337846 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.337966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.340424 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.340434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.340443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.340462 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.341899 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.342379 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.342389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.342391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.342472 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.342753 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.342757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.342758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.343154 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.343349 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.343523 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.343657 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":5504,"thread_start_us":144,"threads_started":1}
I20260620 06:11:56.344334 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.332725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:38960)
0620 06:11:56.333207 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.333263 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.334308 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.334842 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.334850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.335318 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.335525 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.335539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.336353 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.336357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.337198 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.337206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.337684 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.337688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.337899 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.338431 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.338445 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.339891 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.341793 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.341801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.341809 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.342044 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.342331 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.342334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.342336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.342468 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.342845 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.342851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.343161 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.343325 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.343591 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":47,"thread_start_us":117,"threads_started":1}
I20260620 06:11:56.345790 22006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.188@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
I20260620 06:11:56.354638 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.346601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:42980)
0620 06:11:56.346730 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.346744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.346826 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.347223 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.347225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.347343 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.347422 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.347426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.348371 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.348373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.348951 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.348959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.349481 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.349484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.349565 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.350005 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.350016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.351205 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.352728 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.352730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.352731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.352913 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.353463 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.353466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.353467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.353548 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.354073 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.354077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.354234 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.354429 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.354538 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 06:11:56.354722 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.346723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.14.209.189:34149)
0620 06:11:56.346961 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.346965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.346978 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:56.347013 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.347015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.347120 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.347210 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.347554 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.348226 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.349064 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.349368 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.349465 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.351326 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.351329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.351331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.351350 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.352616 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.353096 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.353101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.353103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.353337 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.353689 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.353693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.353695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.353878 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.353975 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.354479 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.354575 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260620 06:11:56.355479 22006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.188@KRBTEST.COM for oryx/127.14.209.188@KRBTEST.COM
I20260620 06:11:56.362805 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.355865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60376)
0620 06:11:56.355981 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.355995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.356080 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.356254 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.356256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.356370 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.356431 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.356435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.357096 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.357098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.357582 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.357588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.357969 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.357972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.358024 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.358452 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.358462 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.359621 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.361678 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.361681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.361682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.361857 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.362103 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.362105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.362106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.362174 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.362465 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.362467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.362533 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.362609 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.362688 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 06:11:56.362869 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.355915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.14.209.188:43821)
0620 06:11:56.356004 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.356007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.356036 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:11:56.356136 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.356137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.356166 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.356226 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.356531 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.357003 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.357665 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.357880 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.357968 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.359769 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.359773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.359775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.359802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.361567 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.361940 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.361942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.361943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.362025 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.362248 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.362250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.362251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.362390 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.362460 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.362661 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.362739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 06:11:56.365890 22006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Bootstrap starting.
I20260620 06:11:56.368141 22006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:56.368827 22006 log.cc:824] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:56.370644 22006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: No bootstrap required, opened a new log
I20260620 06:11:56.373525 22006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } }
I20260620 06:11:56.373725 22006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:56.373809 22006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138f268bef6d450e85d5b44f59531657, State: Initialized, Role: FOLLOWER
I20260620 06:11:56.374210 22006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } }
I20260620 06:11:56.374615 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } } }
I20260620 06:11:56.374722 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:56.375321 22006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:11:56.379653 22023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:56.379727 22023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:56.382979 21854 master_runner.cc:428] Detected that this master 138f268bef6d450e85d5b44f59531657 is joining an existing cluster
I20260620 06:11:56.383031 21854 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.209.188:43821
I20260620 06:11:56.394412 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.385904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.14.209.188:43821)
0620 06:11:56.386002 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.386006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.387280 (+ 1274us) server_negotiation.cc:407] Connection header received
0620 06:11:56.387412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.387415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.387448 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.387506 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.388025 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.388846 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389927 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.390241 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.390310 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.391551 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.391554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.391556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.391576 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.392833 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.393273 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.393275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.393276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.393341 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.393622 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.393624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.393625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.393770 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.393843 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.394135 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.394223 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":39}
I20260620 06:11:56.394521 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.385853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60390)
0620 06:11:56.387209 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.387222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.387326 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.387611 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.387614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.387816 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.387914 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.387920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.388957 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.388960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.389809 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.389819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.390433 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.390436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.390534 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.390938 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.390952 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.391412 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.392932 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.392936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.392938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.393179 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.393424 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.393427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.393429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.393526 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.393857 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.393860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.393948 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.394145 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.394294 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":26,"thread_start_us":70,"threads_started":1}
I20260620 06:11:56.397358 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21350
I20260620 06:11:56.404522 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:39157
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:46813
--webserver_port=33761
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:11:56.407794 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.385158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.14.209.190:46813)
0620 06:11:56.385267 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.385272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.385993 (+ 721us) server_negotiation.cc:407] Connection header received
0620 06:11:56.386170 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.386173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.386224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.386308 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.386756 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.387540 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389175 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.389499 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389577 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.404185 (+ 14608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.404189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.404192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.404217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.406013 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406520 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.406523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.406525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.406607 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406888 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.406891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.406893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.407115 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.407184 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.407533 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.407643 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:11:56.408236 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.385629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.14.209.189:34149)
0620 06:11:56.385727 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.385732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.386189 (+ 457us) server_negotiation.cc:407] Connection header received
0620 06:11:56.386354 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.386357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.386402 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.386452 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.387079 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.387784 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389168 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.389525 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389590 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.403623 (+ 14033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.403629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.403635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.403668 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.406082 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406653 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.406659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.406661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.406749 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.407143 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.407146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.407149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.407388 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.407468 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.407835 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.407991 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 06:11:56.408771 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.385117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:38972)
0620 06:11:56.385901 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.385924 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.386057 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.386322 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.386324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.386535 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.386642 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.386648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.387925 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.387928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.389046 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.389051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.389743 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.389932 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.403651 (+ 13719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.403667 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.404035 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.406132 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.406137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.406400 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.406712 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.406717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.406795 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.407253 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.407256 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.407339 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.408249 (+ 910us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.408571 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":206,"thread_start_us":108,"threads_started":1}
I20260620 06:11:56.408979 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.385545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:42990)
0620 06:11:56.386106 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.386119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.386251 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.386599 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.386602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.386808 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.386936 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.386943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.387888 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.387894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.389032 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.389042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.389743 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.389746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.389927 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.390558 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.390578 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.403497 (+ 12919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.406218 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.406226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.406534 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.406868 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.406872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.406877 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.406999 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.407545 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.407551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.407653 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.408430 (+ 777us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.408649 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":220,"thread_start_us":84,"threads_started":1}
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.190@KRBTEST.COM for oryx/127.14.209.188@KRBTEST.COM
I20260620 06:11:56.422463 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.412863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60392)
0620 06:11:56.413132 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.413148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.413234 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.413879 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.413885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.414099 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.414227 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.414237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.415387 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.415393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.416359 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.416370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.417035 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.417039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.417131 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.417596 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.417610 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.419163 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.421016 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.421019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.421021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.421251 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.421523 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.421526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.421528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.421605 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.422007 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.422010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.422095 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.422183 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.422311 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 06:11:56.422984 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.413384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.14.209.188:43821)
0620 06:11:56.413505 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.413510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.413525 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:56.413592 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.413597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.413659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.413742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.414374 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.415247 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.416502 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.416836 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.416932 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.419304 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.419308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.419311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.419333 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.420881 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.421353 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.421357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.421358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.421430 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.421695 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.421698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.421700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.421873 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.421966 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.422743 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.422843 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:11:56.423877 21288 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.209.188:43821: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "138f268bef6d450e85d5b44f59531657"
member_type: NON_VOTER
last_known_addr {
host: "127.14.209.188"
port: 43821
}
attrs {
promote: true
}
}
dest_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c"
cas_config_opid_index: 5
I20260620 06:11:56.424479 21288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } }
I20260620 06:11:56.425491 21848 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 138f268bef6d450e85d5b44f59531657 (127.14.209.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.426838 21848 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.209.188:43821
I20260620 06:11:56.426986 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.427112 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:56.427803 21980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:56.429136 21980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Refusing update from remote peer 9f5583e026484a63a4eaeaf5dc5afc4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:11:56.429744 21854 master.cc:537] Master@127.14.209.188:43821 shutting down...
I20260620 06:11:56.430555 21848 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:11:56.433071 21275 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c -> Peer 138f268bef6d450e85d5b44f59531657 (127.14.209.188:43821): Couldn't send request to peer 138f268bef6d450e85d5b44f59531657. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:11:56.435491 21854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:56.435751 21854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:56.435837 21854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: stopping tablet replica
I20260620 06:11:56.438225 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.427439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:42992)
0620 06:11:56.427712 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.427727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.427794 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.431428 (+ 3634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.431431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.431452 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:56.431555 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.431561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.434016 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.434019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.435610 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.435621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.437835 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.437838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.437930 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.437975 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.438044 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.438110 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260620 06:11:56.438894 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.429534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.14.209.189:34149)
0620 06:11:56.430170 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.430177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.430194 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:11:56.430245 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.430250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.430317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.430414 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:56.431683 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.432440 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.435749 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.436549 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.436637 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.436754 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.438708 (+ 1954us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.438784 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549}
I20260620 06:11:56.439489 21898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 LEARNER]: Refusing update from remote peer 9f5583e026484a63a4eaeaf5dc5afc4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:11:56.439874 21848 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 06:11:56.441359 21854 master.cc:552] Master@127.14.209.188:43821 shutdown complete.
I20260620 06:11:56.441444 21854 master_runner.cc:305] Clearing existing system tablet
I20260620 06:11:56.441700 21854 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.441787 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.441563 21898 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 138f268bef6d450e85d5b44f59531657 (127.14.209.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.443045 21854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:56.443473 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5583e026484a63a4eaeaf5dc5afc4c. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.443581 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:11:56.443666 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: NON_VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: true } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.443742 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:11:56.444060 22036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.444276 21854 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.444379 21854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:11:56.444416 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
uuid: "138f268bef6d450e85d5b44f59531657"
format_stamp: "Formatted at 2026-06-20 06:11:56 on dist-test-slave-zt01"
server_key: "90893a82bd1ab1e7b3e2626f1a2bb57f"
server_key_iv: "191aab9d08598e27249c3abb107c7da5"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.444887 21848 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: attempting to promote NON_VOTER a9940460356c4d95a1242f7b59d82633 to VOTER
I20260620 06:11:56.445258 21848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } }
I20260620 06:11:56.446324 21854 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:56.446733 21898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 LEARNER]: Refusing update from remote peer 9f5583e026484a63a4eaeaf5dc5afc4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 06:11:56.447094 21847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 06:11:56.448091 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.446375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60406)
0620 06:11:56.447955 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.448026 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1513}
I20260620 06:11:56.448733 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.448522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60416)
0620 06:11:56.448636 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.448676 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 06:11:56.448838 21275 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c -> Peer 138f268bef6d450e85d5b44f59531657 (127.14.209.188:43821): Couldn't send request to peer 138f268bef6d450e85d5b44f59531657. Status: Network error: Client connection negotiation failed: client connection to 127.14.209.188:43821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 06:11:56.449246 21854 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:11:56.449398 21854 log.cc:1196] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 06:11:56.449702 21854 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Deleting consensus metadata
I20260620 06:11:56.449788 21847 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a9940460356c4d95a1242f7b59d82633 (127.14.209.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.449936 21854 master_runner.cc:315] Copying system tablet from 127.14.209.190:46813
I20260620 06:11:56.451129 21854 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: tablet copy: Beginning tablet copy session from remote peer at address 127.14.209.190:46813
I20260620 06:11:56.451216 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.451320 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:56.451733 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.451836 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:56.452013 21898 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a9940460356c4d95a1242f7b59d82633 (127.14.209.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.453522 22035 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.453629 22035 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:56.463344 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.452489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:38974)
0620 06:11:56.455303 (+ 2814us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.455317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.455424 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.455724 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.455784 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.456073 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.456184 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.456191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.457209 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.457213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.458064 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.458073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.458647 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.458651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.458731 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.459281 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.459295 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.459689 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.461801 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.461804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.461807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.462049 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.462329 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.462332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.462334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.462423 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.462807 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.462810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.462908 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.463009 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.463168 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2749,"thread_start_us":86,"threads_started":1}
I20260620 06:11:56.463376 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.452289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.14.209.190:46813)
0620 06:11:56.452407 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.452412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.455404 (+ 2992us) server_negotiation.cc:407] Connection header received
0620 06:11:56.455515 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.455518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.455562 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.455644 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.456314 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.457093 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.458189 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.458527 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.458627 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.459834 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.459838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.459841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.459865 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.461691 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.462147 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.462150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.462152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.462234 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.462514 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.462518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.462519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.462711 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.462805 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.463137 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.463242 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:11:56.464058 21308 tablet_copy_service.cc:140] P 9f5583e026484a63a4eaeaf5dc5afc4c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 138f268bef6d450e85d5b44f59531657 ({username='oryx', principal='oryx/127.14.209.188@KRBTEST.COM'} at 127.0.0.1:38974)
I20260620 06:11:56.464145 21308 tablet_copy_service.cc:161] P 9f5583e026484a63a4eaeaf5dc5afc4c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 138f268bef6d450e85d5b44f59531657 at {username='oryx', principal='oryx/127.14.209.188@KRBTEST.COM'} at 127.0.0.1:38974: session id = 138f268bef6d450e85d5b44f59531657-00000000000000000000000000000000
I20260620 06:11:56.465070 21308 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:11:56.466642 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:11:56.468731 21854 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: tablet copy: Starting download of 0 data blocks...
I20260620 06:11:56.468906 21854 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: tablet copy: Starting download of 1 WAL segments...
I20260620 06:11:56.470291 21854 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:11:56.471823 21308 tablet_copy_service.cc:342] P 9f5583e026484a63a4eaeaf5dc5afc4c: Request end of tablet copy session 138f268bef6d450e85d5b44f59531657-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.209.188@KRBTEST.COM'} at 127.0.0.1:38974
I20260620 06:11:56.471901 21308 tablet_copy_service.cc:433] P 9f5583e026484a63a4eaeaf5dc5afc4c: ending tablet copy session 138f268bef6d450e85d5b44f59531657-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 138f268bef6d450e85d5b44f59531657
I20260620 06:11:56.478668 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:56.480191 22045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.480355 22048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.480581 22046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.480778 21854 server_base.cc:1034] running on GCE node
I20260620 06:11:56.480926 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:56.481168 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:56.482306 21854 hybrid_clock.cc:648] HybridClock initialized: now 1781935916482292 us; error 27 us; skew 500 ppm
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:56.483898 21854 init.cc:377] Logged in from keytab as oryx/127.14.209.188@KRBTEST.COM (short username oryx)
I20260620 06:11:56.484328 21854 webserver.cc:460] Webserver started at http://127.14.209.188:35157/ using document root <none> and password file <none>
I20260620 06:11:56.484500 21854 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.484534 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.485718 21854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:11:56.486658 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.486907 21854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:11:56.486979 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
uuid: "138f268bef6d450e85d5b44f59531657"
format_stamp: "Formatted at 2026-06-20 06:11:56 on dist-test-slave-zt01"
server_key: "90893a82bd1ab1e7b3e2626f1a2bb57f"
server_key_iv: "191aab9d08598e27249c3abb107c7da5"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.487046 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.499432 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:56.501433 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:56.506524 21854 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.188:43821
I20260620 06:11:56.506623 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.188:43821 every 8 connection(s)
I20260620 06:11:56.507328 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
I20260620 06:11:56.509125 22108 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:11:56.510149 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Bootstrap starting.
I20260620 06:11:56.521343 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:11:56.521639 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: Bootstrap complete.
I20260620 06:11:56.522194 22108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } }
I20260620 06:11:56.522337 22108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 138f268bef6d450e85d5b44f59531657, State: Initialized, Role: LEARNER
I20260620 06:11:56.522455 22108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } }
I20260620 06:11:56.522786 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.522907 22108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:56.522966 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:11:56.527357 22122 catalog_manager.cc:1259] Loaded cluster ID: e373f05ea2c3438e82bb64d3fba9d0ad
I20260620 06:11:56.527411 22122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: loading cluster ID for follower catalog manager: success
I20260620 06:11:56.529471 22122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: acquiring CA information for follower catalog manager: success
I20260620 06:11:56.530846 22122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 06:11:56.543011 22032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:56.543274 22032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:56.543349 22032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:56.547261 22032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:56.547307 22032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:56.547394 22032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:11:56.552584 22032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:39157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=33761
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:56.553504 22032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:56.554428 22032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:56.563660 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.563660 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.563760 22032 server_base.cc:1034] running on GCE node
W20260620 06:11:56.563649 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.564179 22032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:56.564770 22032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:56.565974 22032 hybrid_clock.cc:648] HybridClock initialized: now 1781935916565965 us; error 32 us; skew 500 ppm
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:56.569051 22032 init.cc:377] Logged in from keytab as oryx/127.14.209.129@KRBTEST.COM (short username oryx)
I20260620 06:11:56.569844 22032 webserver.cc:460] Webserver started at http://127.14.209.129:33761/ using document root <none> and password file <none>
I20260620 06:11:56.570382 22032 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.570533 22032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.574065 22032 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:56.576073 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.577487 22032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:56.577615 22032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "49d206ba20aa4807b8070b2ef009df51"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "802d1273d289509a3bd10ca19251ada0"
server_key_iv: "ef22dfa62bdf699c0cbb17c4c82585a9"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.578105 22032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.607802 22032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:56.610335 22032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:56.610535 22032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:56.611112 22032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:56.611876 22032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:56.611943 22032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.612010 22032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:56.612061 22032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.617877 21613 heartbeater.cc:503] Master 127.14.209.190:46813 was elected leader, sending a full tablet report...
I20260620 06:11:56.621366 22032 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:39157
I20260620 06:11:56.621376 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:39157 every 8 connection(s)
I20260620 06:11:56.622244 22032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:11:56.631659 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 22032
I20260620 06:11:56.631856 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21484
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.188@KRBTEST.COM
I20260620 06:11:56.638599 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.623935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:41655 (local address 127.14.209.190:46813)
0620 06:11:56.624053 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.624059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.625105 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 06:11:56.626132 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.626135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.626202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.626279 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.627857 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.628655 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.629912 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.630237 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.630328 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.634056 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.634063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.634069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.634119 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.636233 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.636886 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.636899 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.636901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.636988 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637375 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.637378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.637379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.637583 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.637656 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.638332 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.638440 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:11:56.638705 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.624012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:35923 (local address 127.14.209.189:34149)
0620 06:11:56.624171 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.624174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.625105 (+ 931us) server_negotiation.cc:407] Connection header received
0620 06:11:56.626954 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.626960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.627017 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.627131 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.627851 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.628369 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.629944 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.630158 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.630273 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.634479 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.634486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.634490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.634520 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.636346 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.636950 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.636956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.636961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.637043 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637378 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.637384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.637389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.637622 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.637696 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.638339 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.638471 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 06:11:56.639503 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.625529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:52227 (local address 127.14.209.188:43821)
0620 06:11:56.625749 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.625753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.626013 (+ 260us) server_negotiation.cc:407] Connection header received
0620 06:11:56.626199 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.626202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.626254 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.626332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.627816 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.628877 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.629760 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.630121 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.630222 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.634514 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.634520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.634523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.634545 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.636528 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637021 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.637027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.637030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.637112 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637380 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.637383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.637385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.637583 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.637671 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.639178 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.639324 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260620 06:11:56.639878 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.625443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.14.209.129:52227)
0620 06:11:56.625910 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.625925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.626023 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.626538 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.626546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.627131 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.627683 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.627695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.628979 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.628984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.629641 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.629649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.630266 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.630270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.630436 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.631264 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.631275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.634362 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.636645 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.636647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.636649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.636918 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.637209 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.637213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.637283 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.637723 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.637729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.638062 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.638262 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.638554 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":140,"spinlock_wait_cycles":80000,"thread_start_us":74,"threads_started":1}
I20260620 06:11:56.639878 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.624441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:41655)
0620 06:11:56.624957 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.625013 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.625898 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.626541 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.626549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.627130 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.627712 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.627720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.628775 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.628778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.629794 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.629803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.630441 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.630443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.630531 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.631339 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.631351 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.633846 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.636367 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.636373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.636386 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.636770 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.637096 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.637101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.637265 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.637723 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.637729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.638062 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.638765 (+ 703us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.638884 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":238,"spinlock_wait_cycles":64768,"thread_start_us":131,"threads_started":1}
I20260620 06:11:56.640805 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.624441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.14.209.129:35923)
0620 06:11:56.624957 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.625013 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.626828 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.627142 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.627144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.627312 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.627712 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.627720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.628503 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.628505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.629795 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.629803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.630383 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.630386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.630478 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.631234 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.631256 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.634362 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.636471 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.636478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.636483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.636826 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.637154 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.637160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.637165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.637278 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.637768 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.637774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.638073 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.640358 (+ 2285us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.640524 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":173,"spinlock_wait_cycles":83968,"thread_start_us":127,"threads_started":1}
I20260620 06:11:56.640931 22252 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46813
I20260620 06:11:56.641289 22252 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:56.641852 22252 heartbeater.cc:511] Master 127.14.209.190:46813 requested a full tablet report, sending...
I20260620 06:11:56.641873 22251 heartbeater.cc:347] Connected to a master server at 127.14.209.188:43821
I20260620 06:11:56.641942 22251 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:56.642071 22251 heartbeater.cc:511] Master 127.14.209.188:43821 requested a full tablet report, sending...
I20260620 06:11:56.642242 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:42033
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:46813
--webserver_port=46741
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:11:56.642329 22253 heartbeater.cc:347] Connected to a master server at 127.14.209.189:34149
I20260620 06:11:56.642393 22253 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:56.642508 22253 heartbeater.cc:511] Master 127.14.209.189:34149 requested a full tablet report, sending...
I20260620 06:11:56.643081 21288 ts_manager.cc:194] Re-registered known tserver with Master: 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129:39157)
I20260620 06:11:56.643431 22072 ts_manager.cc:194] Registered new tserver with Master: 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129:39157)
I20260620 06:11:56.643632 21888 ts_manager.cc:194] Registered new tserver with Master: 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129:39157)
I20260620 06:11:56.643750 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:41655
W20260620 06:11:56.748775 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:56.749069 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:56.749140 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:56.753358 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:56.753407 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:56.753461 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:11:56.758534 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:42033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=46741
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:56.759502 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:56.760738 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:56.770305 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.770335 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.770335 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.770634 22259 server_base.cc:1034] running on GCE node
I20260620 06:11:56.771030 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:56.771682 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:56.772862 22259 hybrid_clock.cc:648] HybridClock initialized: now 1781935916772836 us; error 54 us; skew 500 ppm
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:56.775754 22259 init.cc:377] Logged in from keytab as oryx/127.14.209.130@KRBTEST.COM (short username oryx)
I20260620 06:11:56.776656 22259 webserver.cc:460] Webserver started at http://127.14.209.130:46741/ using document root <none> and password file <none>
I20260620 06:11:56.777220 22259 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.777323 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.781025 22259 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 06:11:56.782949 22274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.784116 22259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:56.784245 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "c079e9d31932b6f30cabbcc39a6115d7"
server_key_iv: "a17bf6e1bfcf65332fb39d1600d503b1"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.784761 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:56.803426 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:56.806370 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:56.806546 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:56.807112 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:56.807891 22259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:56.807960 22259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.808025 22259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:56.808077 22259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.817040 22259 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:42033
I20260620 06:11:56.817068 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:42033 every 8 connection(s)
I20260620 06:11:56.817530 21747 heartbeater.cc:503] Master 127.14.209.190:46813 was elected leader, sending a full tablet report...
I20260620 06:11:56.818009 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:11:56.818658 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 22259
I20260620 06:11:56.818847 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21618
I20260620 06:11:56.827549 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:39817
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:46813
--webserver_port=42141
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.188@KRBTEST.COM
I20260620 06:11:56.836541 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.820047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:36791 (local address 127.14.209.188:43821)
0620 06:11:56.820199 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.820204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.821686 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 06:11:56.822744 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.822750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.822813 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.822914 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.824685 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.825489 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.827549 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.827928 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.827990 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.832734 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.832738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.832742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.832772 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.834709 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.835227 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.835231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.835232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.835315 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.835653 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.835656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.835658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.835875 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.835950 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.836267 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.836386 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:11:56.837050 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.820465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.14.209.130:36791)
0620 06:11:56.821547 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.821597 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.822524 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.823116 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.823126 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.823759 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.824526 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.824544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.825667 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.825670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.827415 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.827425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.828317 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.828320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.828473 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.829321 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.829337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.832573 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.834818 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.834822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.834824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.835110 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.835456 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.835459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.835461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.835545 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.836004 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.836006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.836091 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.836671 (+ 580us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.836798 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":346,"spinlock_wait_cycles":11776,"thread_start_us":129,"threads_started":1}
I20260620 06:11:56.837438 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.821589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:57759 (local address 127.14.209.189:34149)
0620 06:11:56.821720 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.821727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.823004 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 06:11:56.823186 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.823189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.823241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.823299 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.824683 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.825491 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.827549 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.827887 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.827971 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.831443 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.831449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.831452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.831476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.833253 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.833870 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.833873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.833876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.833964 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.834885 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.834892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.834895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.835156 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.835248 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.837163 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.837278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:56.837050 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.821459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.14.209.130:57759)
0620 06:11:56.822927 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.822940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.823081 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.823463 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.823466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.823803 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.824526 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.824544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.825667 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.825670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.827415 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.827425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.828170 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.828175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.828469 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.829254 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.829277 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.831233 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.833378 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.833384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.833399 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.833776 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.834065 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.834069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.834074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.834269 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.835300 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.835306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.835604 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.835787 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.836098 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1383,"mutex_wait_us":481,"spinlock_wait_cycles":33920,"thread_start_us":68,"threads_started":1}
I20260620 06:11:56.838572 22388 heartbeater.cc:347] Connected to a master server at 127.14.209.188:43821
I20260620 06:11:56.838657 22390 heartbeater.cc:347] Connected to a master server at 127.14.209.189:34149
I20260620 06:11:56.838789 22388 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:56.838788 22390 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:56.839031 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.820465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:55039)
0620 06:11:56.821748 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.821759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.822524 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.823366 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.823372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.823802 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:56.824527 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.824545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.825738 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.825744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.827404 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.827415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.828182 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.828188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.828448 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.829431 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:56.829449 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:56.832225 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:56.835435 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.835441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.835445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.836309 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.836927 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:56.836931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:56.836935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:56.837036 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:56.838455 (+ 1419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:56.838461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:56.838573 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.838726 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.838859 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":334,"spinlock_wait_cycles":62208,"thread_start_us":140,"threads_started":1}
I20260620 06:11:56.839118 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.820625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:55039 (local address 127.14.209.190:46813)
0620 06:11:56.820774 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.820780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.821821 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 06:11:56.823077 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.823083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.823160 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.823238 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:56.824682 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.825590 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.827541 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.827920 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.828015 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.832641 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:56.832647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:56.832654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:56.832688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:56.835258 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.836449 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.836455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.836460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.836553 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:56.838023 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:56.838029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:56.838032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:56.838288 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:56.838370 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.838823 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.838954 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:56.839354 22390 heartbeater.cc:511] Master 127.14.209.189:34149 requested a full tablet report, sending...
I20260620 06:11:56.839366 22388 heartbeater.cc:511] Master 127.14.209.188:43821 requested a full tablet report, sending...
I20260620 06:11:56.840247 22389 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46813
I20260620 06:11:56.840318 22389 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:56.840394 21888 ts_manager.cc:194] Registered new tserver with Master: c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033)
I20260620 06:11:56.840448 22389 heartbeater.cc:511] Master 127.14.209.190:46813 requested a full tablet report, sending...
I20260620 06:11:56.840857 22072 ts_manager.cc:194] Registered new tserver with Master: c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033)
I20260620 06:11:56.841096 21288 ts_manager.cc:194] Re-registered known tserver with Master: c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033)
I20260620 06:11:56.841856 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:55039
W20260620 06:11:56.937791 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:56.938020 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:56.938060 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:56.941851 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:56.941900 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:56.941959 22395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:11:56.947108 22395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:39817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=42141
--tserver_master_addrs=127.14.209.190:46813,127.14.209.189:34149,127.14.209.188:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:56.948031 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:56.949012 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:56.959112 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.959118 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:56.959215 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:56.959635 22395 server_base.cc:1034] running on GCE node
I20260620 06:11:56.960034 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:56.960736 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:56.961812 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.957528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.14.209.188:43821)
0620 06:11:56.957658 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:56.957665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:56.957681 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:56.957738 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:56.957744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:56.957813 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:56.957908 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:56.958365 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.959121 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.960762 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.961376 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.961503 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.961632 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:56.961665 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 06:11:56.961719 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:56.961818 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:56.956027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60418)
0620 06:11:56.956214 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:56.956228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:56.956323 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:56.958086 (+ 1763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:56.958090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:56.958116 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:56.958231 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.958238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.959273 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.959277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.960607 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:56.960621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:56.961497 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:56.961500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:56.961567 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:56.961597 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:11:56.961650 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 06:11:56.961707 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260620 06:11:56.962051 22395 hybrid_clock.cc:648] HybridClock initialized: now 1781935916962018 us; error 46 us; skew 500 ppm
I20260620 06:11:56.962395 22082 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 06:11:56.962761 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5583e026484a63a4eaeaf5dc5afc4c. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:56.962908 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: This master's current role is: LEARNER
Jun 20 06:11:56 dist-test-slave-zt01 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:56.964797 22395 init.cc:377] Logged in from keytab as oryx/127.14.209.131@KRBTEST.COM (short username oryx)
I20260620 06:11:56.965554 22395 webserver.cc:460] Webserver started at http://127.14.209.131:42141/ using document root <none> and password file <none>
I20260620 06:11:56.966068 22395 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:56.966163 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:56.970501 22395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:56.972708 22412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:56.973930 22395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:56.974068 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "aee8328a367745b286e8981448426b46"
format_stamp: "Formatted at 2026-06-20 06:11:55 on dist-test-slave-zt01"
server_key: "165250be7e1dbf170527568913a29e9b"
server_key_iv: "1d0c528e9a9ca2391adf825476477fae"
server_key_version: "encryptionkey@0"
I20260620 06:11:56.974574 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:57.007588 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:57.010201 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:57.010381 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:57.010903 22395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:11:57.011701 22395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:11:57.011770 22395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:57.011833 22395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:11:57.011881 22395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:57.020670 22395 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:39817
I20260620 06:11:57.020695 22525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:39817 every 8 connection(s)
I20260620 06:11:57.021559 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:11:57.024292 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 22395
Jun 20 06:11:57 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
Jun 20 06:11:57 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.188@KRBTEST.COM
I20260620 06:11:57.038144 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.026155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.14.209.189:34149)
0620 06:11:57.026707 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.026714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.027088 (+ 374us) server_negotiation.cc:407] Connection header received
0620 06:11:57.027127 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.027129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.027173 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.027245 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.028209 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.028968 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.030128 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.030483 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.030569 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.032359 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.032363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.032365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.032389 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.034379 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.036469 (+ 2090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.036473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.036475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.036576 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.036882 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.036885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.036887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.037103 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.037218 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.037824 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.037947 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":87,"threads_started":1}
I20260620 06:11:57.039268 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.025664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:34491 (local address 127.14.209.188:43821)
0620 06:11:57.025787 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.025792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.025805 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:57.025956 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.025960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.026008 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.026064 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.028198 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.028950 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.031174 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.031556 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.031627 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.034710 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.034713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.034716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.034739 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.037051 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.037624 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.037627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.037629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.037715 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.038108 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.038111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.038112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.038315 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.038383 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.038993 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.039146 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 06:11:57 dist-test-slave-zt01 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935916, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.189@KRBTEST.COM
I20260620 06:11:57.040725 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.025124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.14.209.131:34491)
0620 06:11:57.025726 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.025740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.025823 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.026438 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.026447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.027107 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:57.028070 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.028082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.029093 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.029096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.031002 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.031021 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.031768 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.031771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.031872 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.032385 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:57.032400 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:57.034542 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:57.037170 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:57.037176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:57.037188 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:57.037515 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:57.037829 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:57.037832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:57.037834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:57.037989 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:57.038462 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:57.038469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:57.038734 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.039418 (+ 684us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.039739 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":1512,"thread_start_us":74,"threads_started":1}
I20260620 06:11:57.041366 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.023654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:50317 (local address 127.14.209.190:46813)
0620 06:11:57.023773 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.023777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.025426 (+ 1649us) server_negotiation.cc:407] Connection header received
0620 06:11:57.025958 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.025961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.026023 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.026074 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.028058 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.028867 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.030495 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.030800 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.030880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.035559 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.035562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.035565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.035591 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.037606 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.039867 (+ 2261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.039870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.039872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.039955 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.040245 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.040248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.040250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.040434 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.040523 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.041071 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.041190 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":489216}
I20260620 06:11:57.041676 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.024002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:50317)
0620 06:11:57.025339 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.025353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.025828 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.026438 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.026447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.027108 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:57.027873 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.027891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.029051 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.029057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.030371 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.030380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.031107 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.031114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.031334 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.031933 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:57.031959 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:57.035410 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:57.039420 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:57.039423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:57.039425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:57.039754 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:57.040065 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:57.040068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:57.040070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:57.040150 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:57.040580 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:57.040582 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:57.040660 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.040907 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.041037 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":288,"spinlock_wait_cycles":30464,"thread_start_us":165,"threads_started":1}
I20260620 06:11:57.041973 22526 heartbeater.cc:347] Connected to a master server at 127.14.209.188:43821
I20260620 06:11:57.042208 22526 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:57.042726 22526 heartbeater.cc:511] Master 127.14.209.188:43821 requested a full tablet report, sending...
I20260620 06:11:57.042858 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.026790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.14.209.188:43821)
0620 06:11:57.028192 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.028196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.028574 (+ 378us) server_negotiation.cc:407] Connection header received
0620 06:11:57.028610 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.028612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.028650 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.028719 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.030098 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.030872 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.032550 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.032928 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.033021 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.035762 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.035768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.035774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.035807 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.038870 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.040868 (+ 1998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.040874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.040877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.040952 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.041250 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.041254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.041255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.041481 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.041616 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.042587 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.042703 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":81,"threads_started":1}
I20260620 06:11:57.043301 22527 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46813
I20260620 06:11:57.043391 22527 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:57.043594 22527 heartbeater.cc:511] Master 127.14.209.190:46813 requested a full tablet report, sending...
I20260620 06:11:57.044050 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.027481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.14.209.190:46813)
0620 06:11:57.028056 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.028060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.029041 (+ 981us) server_negotiation.cc:407] Connection header received
0620 06:11:57.029084 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.029087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.029136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.029217 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.033771 (+ 4554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.034541 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.035709 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.036005 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.036129 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.037922 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.037926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.037930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.037956 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.040226 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.040698 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.040702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.040703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.041726 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.042596 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.042599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.042601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.042844 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.043190 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.043758 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.043867 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":97,"threads_started":1}
I20260620 06:11:57.044286 21288 ts_manager.cc:194] Re-registered known tserver with Master: aee8328a367745b286e8981448426b46 (127.14.209.131:39817)
I20260620 06:11:57.044303 22072 ts_manager.cc:194] Registered new tserver with Master: aee8328a367745b286e8981448426b46 (127.14.209.131:39817)
I20260620 06:11:57.045179 21288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:50317
I20260620 06:11:57.045893 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.023576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:38905 (local address 127.14.209.189:34149)
0620 06:11:57.023720 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.023725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.024762 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 06:11:57.025971 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.025976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.026054 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.026111 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.029359 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.030137 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.034070 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.034462 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.034538 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.042541 (+ 8003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.042544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.042548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.042573 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.044205 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.044653 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.044656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.044658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.044733 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.045024 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.045028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.045030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.045219 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.045305 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.045657 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.045762 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:57.045898 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.024002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.14.209.131:38905)
0620 06:11:57.024563 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.024664 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.025744 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.026438 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.026448 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.027223 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:57.027871 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.027891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.032999 (+ 5108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.033002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.033929 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.033940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.034732 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.034735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.034840 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.039693 (+ 4853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:57.039710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:57.042410 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:57.044320 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:57.044323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:57.044325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:57.044542 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:57.044832 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:57.044835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:57.044836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:57.044908 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:57.045350 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:57.045355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:57.045460 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.045592 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.045723 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":53,"thread_start_us":677,"threads_started":1}
I20260620 06:11:57.046408 22528 heartbeater.cc:347] Connected to a master server at 127.14.209.189:34149
I20260620 06:11:57.046490 22528 heartbeater.cc:464] Registering TS with master...
I20260620 06:11:57.046629 22528 heartbeater.cc:511] Master 127.14.209.189:34149 requested a full tablet report, sending...
I20260620 06:11:57.047411 21888 ts_manager.cc:194] Registered new tserver with Master: aee8328a367745b286e8981448426b46 (127.14.209.131:39817)
I20260620 06:11:57.047881 21288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:11:57.050937 21288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:11:57.053238 22404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: NON_VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: true } } }
I20260620 06:11:57.053383 22404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:57.056149 22404 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: attempting to promote NON_VOTER 138f268bef6d450e85d5b44f59531657 to VOTER
I20260620 06:11:57.056421 22404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:11:57.058811 22081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 LEARNER]: Refusing update from remote peer 9f5583e026484a63a4eaeaf5dc5afc4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 06:11:57.059136 21898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Refusing update from remote peer 9f5583e026484a63a4eaeaf5dc5afc4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 06:11:57.059183 22544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 06:11:57.059480 22544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 06:11:57.061015 22545 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 138f268bef6d450e85d5b44f59531657 (127.14.209.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.061499 22081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 138f268bef6d450e85d5b44f59531657 (127.14.209.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.061754 21898 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 138f268bef6d450e85d5b44f59531657 (127.14.209.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.062495 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.062595 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:57.062708 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.062784 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:57.062878 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.062983 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:57.063433 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.063534 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:57.063828 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:11:57.063956 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:57.070098 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.064938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:42033 (local address 127.0.0.1:60784)
0620 06:11:57.065501 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.065513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.065603 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.066303 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.066306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.066324 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:57.066451 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.066458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.067887 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.067890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.068627 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.068659 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.069740 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.069744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.069824 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.069844 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.069907 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.069971 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260620 06:11:57.070160 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.065500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.14.209.130:42033)
0620 06:11:57.065817 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.065830 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.065854 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:11:57.065915 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.065919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.066151 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.066320 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:57.066617 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.067748 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.068795 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.069616 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.069753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.069878 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.069931 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.070029 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260620 06:11:57.070559 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.064784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:39817 (local address 127.0.0.1:60994)
0620 06:11:57.064939 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.064954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.065048 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.066317 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.066319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.066337 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:57.066444 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.066452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.067860 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.067864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.069104 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.069114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.070186 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.070189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.070271 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.070291 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.070341 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.070409 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":62}
I20260620 06:11:57.070873 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.065405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.14.209.129:39157)
0620 06:11:57.065698 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.065710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.066631 (+ 921us) server_negotiation.cc:407] Connection header received
0620 06:11:57.066741 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.066745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.066878 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.067029 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:57.067224 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.067969 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.069350 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.070069 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.070181 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.070291 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.070628 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.070733 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":120,"threads_started":1}
I20260620 06:11:57.071143 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.065240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39157 (local address 127.0.0.1:33836)
0620 06:11:57.066549 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.066561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.066638 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.067004 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.067007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.067023 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:57.067128 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.067135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.068104 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.068108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.069237 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.069244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.070349 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.070352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.070423 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.070441 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.070503 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.070556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"thread_start_us":75,"threads_started":1}
I20260620 06:11:57.071584 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.065182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.14.209.131:39817)
0620 06:11:57.065924 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.065935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.065955 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:11:57.066013 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.066018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.066190 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.066338 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:57.066563 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.067743 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.069239 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.070052 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.070812 (+ 760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.070926 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.071360 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.071451 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":87,"threads_started":1}
I20260620 06:11:57.073181 22322 tablet_service.cc:1467] Processing CreateTablet for tablet 320c00efd76342c7aad83fb4315ffe1e (DEFAULT_TABLE table=test-table [id=6705ae6c166344d48c4c9cc6ab1b5699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:57.073592 22460 tablet_service.cc:1467] Processing CreateTablet for tablet 320c00efd76342c7aad83fb4315ffe1e (DEFAULT_TABLE table=test-table [id=6705ae6c166344d48c4c9cc6ab1b5699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:57.073580 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 320c00efd76342c7aad83fb4315ffe1e (DEFAULT_TABLE table=test-table [id=6705ae6c166344d48c4c9cc6ab1b5699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:11:57.074229 22322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320c00efd76342c7aad83fb4315ffe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:57.074430 22460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320c00efd76342c7aad83fb4315ffe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:57.074432 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320c00efd76342c7aad83fb4315ffe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:57.081624 22559 tablet_bootstrap.cc:492] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Bootstrap starting.
I20260620 06:11:57.082665 22560 tablet_bootstrap.cc:492] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Bootstrap starting.
I20260620 06:11:57.083360 22561 tablet_bootstrap.cc:492] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Bootstrap starting.
I20260620 06:11:57.084319 22559 tablet_bootstrap.cc:654] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:57.084331 22560 tablet_bootstrap.cc:654] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:57.085004 22560 log.cc:824] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:57.085103 22559 log.cc:824] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:57.085196 22561 tablet_bootstrap.cc:654] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:57.085866 22561 log.cc:824] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:57.086927 22560 tablet_bootstrap.cc:492] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: No bootstrap required, opened a new log
I20260620 06:11:57.086959 22559 tablet_bootstrap.cc:492] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: No bootstrap required, opened a new log
I20260620 06:11:57.087139 22560 ts_tablet_manager.cc:1397] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:11:57.087164 22559 ts_tablet_manager.cc:1397] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:11:57.087663 22561 tablet_bootstrap.cc:492] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: No bootstrap required, opened a new log
I20260620 06:11:57.087815 22561 ts_tablet_manager.cc:1397] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:11:57.091006 22559 raft_consensus.cc:348] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.090884 22560 raft_consensus.cc:348] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.091277 22559 raft_consensus.cc:374] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:57.091331 22559 raft_consensus.cc:729] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee8328a367745b286e8981448426b46, State: Initialized, Role: FOLLOWER
I20260620 06:11:57.091337 22560 raft_consensus.cc:374] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:57.091385 22560 raft_consensus.cc:729] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d206ba20aa4807b8070b2ef009df51, State: Initialized, Role: FOLLOWER
I20260620 06:11:57.091293 22561 raft_consensus.cc:348] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.091516 22561 raft_consensus.cc:374] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:57.091557 22561 raft_consensus.cc:729] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7af512cd94d7288ddfdf4a9f2fc8b, State: Initialized, Role: FOLLOWER
I20260620 06:11:57.091778 22559 consensus_queue.cc:260] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.091789 22560 consensus_queue.cc:260] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.091871 22561 consensus_queue.cc:260] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.092700 22559 ts_tablet_manager.cc:1428] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:11:57.092705 22560 ts_tablet_manager.cc:1428] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:11:57.092705 22561 ts_tablet_manager.cc:1428] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:11:57.093210 22389 heartbeater.cc:503] Master 127.14.209.190:46813 was elected leader, sending a full tablet report...
I20260620 06:11:57.093760 22252 heartbeater.cc:503] Master 127.14.209.190:46813 was elected leader, sending a full tablet report...
I20260620 06:11:57.094926 22527 heartbeater.cc:503] Master 127.14.209.190:46813 was elected leader, sending a full tablet report...
W20260620 06:11:57.124156 22254 tablet.cc:2369] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:11:57.138971 22565 raft_consensus.cc:482] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:11:57.139163 22565 raft_consensus.cc:504] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.140168 22565 leader_election.cc:288] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee8328a367745b286e8981448426b46 (127.14.209.131:39817), c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033)
I20260620 06:11:57.145033 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.140641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:47349 (local address 127.14.209.131:39817)
0620 06:11:57.140746 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.140750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.140763 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:57.140809 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.140814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.140859 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.140925 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:57.141167 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.141929 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.143752 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.144361 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.144470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.144520 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.144831 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.144908 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:57.145151 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.140368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:39817 (local address 127.14.209.129:47349)
0620 06:11:57.140514 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.140533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.140613 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.140941 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.140944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.140980 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:57.141064 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.141072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.142046 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.142050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.143630 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.143640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.144493 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.144498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.144670 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.144713 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.144797 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.145008 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":301}
I20260620 06:11:57.145450 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.140500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:54145 (local address 127.14.209.130:42033)
0620 06:11:57.140641 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.140646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.140663 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:11:57.140717 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.140720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.140767 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.140855 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:57.141138 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.141934 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.143752 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.144342 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.144470 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.144520 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.145247 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.145341 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:57.145442 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.140390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:42033 (local address 127.14.209.129:54145)
0620 06:11:57.140574 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:57.140585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:57.140637 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:57.140867 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:57.140869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:57.140881 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:57.141000 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.141007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.142046 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.142050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.143633 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:57.143640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.144493 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.144515 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:57.144665 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.144712 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 06:11:57.144797 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:11:57.144984 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":270}
I20260620 06:11:57.145761 22480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320c00efd76342c7aad83fb4315ffe1e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee8328a367745b286e8981448426b46" is_pre_election: true
I20260620 06:11:57.146029 22480 raft_consensus.cc:2454] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 0.
I20260620 06:11:57.146046 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320c00efd76342c7aad83fb4315ffe1e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" is_pre_election: true
I20260620 06:11:57.146338 22342 raft_consensus.cc:2454] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 0.
I20260620 06:11:57.146485 22141 leader_election.cc:302] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d206ba20aa4807b8070b2ef009df51, aee8328a367745b286e8981448426b46; no voters:
I20260620 06:11:57.146780 22565 raft_consensus.cc:2781] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:11:57.146872 22565 raft_consensus.cc:482] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:11:57.146929 22565 raft_consensus.cc:3037] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:57.147949 22565 raft_consensus.cc:504] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.148288 22565 leader_election.cc:288] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 election: Requested vote from peers aee8328a367745b286e8981448426b46 (127.14.209.131:39817), c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033)
I20260620 06:11:57.148615 22480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320c00efd76342c7aad83fb4315ffe1e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee8328a367745b286e8981448426b46"
I20260620 06:11:57.148653 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320c00efd76342c7aad83fb4315ffe1e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b"
I20260620 06:11:57.148732 22480 raft_consensus.cc:3037] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:57.148797 22342 raft_consensus.cc:3037] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:57.149821 22342 raft_consensus.cc:2454] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 1.
I20260620 06:11:57.149811 22480 raft_consensus.cc:2454] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 1.
I20260620 06:11:57.150103 22138 leader_election.cc:302] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d206ba20aa4807b8070b2ef009df51, c7e7af512cd94d7288ddfdf4a9f2fc8b; no voters:
I20260620 06:11:57.150285 22565 raft_consensus.cc:2781] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:57.150473 22565 raft_consensus.cc:686] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 1 LEADER]: Becoming Leader. State: Replica: 49d206ba20aa4807b8070b2ef009df51, State: Running, Role: LEADER
I20260620 06:11:57.150966 22565 consensus_queue.cc:237] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } }
I20260620 06:11:57.154301 21286 catalog_manager.cc:5495] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129). New cstate: current_term: 1 leader_uuid: "49d206ba20aa4807b8070b2ef009df51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } health_report { overall_health: HEALTHY } } }
I20260620 06:11:57.171329 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.168091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.14.209.129:39157)
0620 06:11:57.168210 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.168219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.168232 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:57.168380 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.168382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.168431 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.168513 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:57.168710 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.169222 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.170021 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.170238 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.170315 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.170555 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:57.170991 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:57.171143 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.171191 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.171236 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:11:57.176545 22342 raft_consensus.cc:1261] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Refusing update from remote peer 49d206ba20aa4807b8070b2ef009df51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:57.176594 22480 raft_consensus.cc:1261] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Refusing update from remote peer 49d206ba20aa4807b8070b2ef009df51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:57.177266 22565 consensus_queue.cc:1035] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:57.177443 22568 consensus_queue.cc:1035] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:57.195679 21287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:57.195850 21287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:11:57.199970 21287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f5583e026484a63a4eaeaf5dc5afc4c: Sending DeleteTablet for 3 replicas of tablet 320c00efd76342c7aad83fb4315ffe1e
I20260620 06:11:57.200742 22322 tablet_service.cc:1514] Processing DeleteTablet for tablet 320c00efd76342c7aad83fb4315ffe1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:57 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:60784
I20260620 06:11:57.200870 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet 320c00efd76342c7aad83fb4315ffe1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:57 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:33836
I20260620 06:11:57.201222 22581 tablet_replica.cc:330] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: stopping tablet replica
I20260620 06:11:57.201555 22581 raft_consensus.cc:2229] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:57.201627 22460 tablet_service.cc:1514] Processing DeleteTablet for tablet 320c00efd76342c7aad83fb4315ffe1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:11:57 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:60994
I20260620 06:11:57.201850 22581 raft_consensus.cc:2258] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:57.202178 22583 tablet_replica.cc:330] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: stopping tablet replica
I20260620 06:11:57.202498 22583 raft_consensus.cc:2229] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:11:57.202802 22583 raft_consensus.cc:2258] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:57.203353 22581 ts_tablet_manager.cc:1905] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:57.204541 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21256
I20260620 06:11:57.205045 22583 ts_tablet_manager.cc:1905] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:11:57.206136 22581 ts_tablet_manager.cc:1918] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:57.206231 22581 log.cc:1196] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/320c00efd76342c7aad83fb4315ffe1e
I20260620 06:11:57.206538 22581 ts_tablet_manager.cc:1939] T 320c00efd76342c7aad83fb4315ffe1e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Deleting consensus metadata
I20260620 06:11:57.208845 22583 ts_tablet_manager.cc:1918] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:57.208945 22583 log.cc:1196] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/320c00efd76342c7aad83fb4315ffe1e
I20260620 06:11:57.209270 22583 ts_tablet_manager.cc:1939] T 320c00efd76342c7aad83fb4315ffe1e P aee8328a367745b286e8981448426b46: Deleting consensus metadata
I20260620 06:11:57.210807 22582 tablet_replica.cc:330] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: stopping tablet replica
I20260620 06:11:57.211128 22582 raft_consensus.cc:2229] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:11:57.211517 22582 raft_consensus.cc:2258] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:11:57.212898 22582 ts_tablet_manager.cc:1905] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:11:57.213838 22276 connection.cc:535] server connection from 127.0.0.1:60784 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 06:11:57.214115 22416 connection.cc:535] server connection from 127.0.0.1:60994 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:11:57.216148 22582 ts_tablet_manager.cc:1918] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:11:57.216241 22582 log.cc:1196] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/320c00efd76342c7aad83fb4315ffe1e
I20260620 06:11:57.216598 22582 ts_tablet_manager.cc:1939] T 320c00efd76342c7aad83fb4315ffe1e P 49d206ba20aa4807b8070b2ef009df51: Deleting consensus metadata
W20260620 06:11:57.217391 22140 connection.cc:441] server connection from 127.0.0.1:33836 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33836 (request call id 1) could send its response
I20260620 06:11:57.221599 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.203583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.14.209.189:34149)
0620 06:11:57.203684 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.203689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.210555 (+ 6866us) server_negotiation.cc:407] Connection header received
0620 06:11:57.210674 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.210677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.210728 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.210798 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.212045 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.212858 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.215308 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.215646 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.215718 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.218073 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.218076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.218079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.218100 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.219772 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.220316 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.220320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.220322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.220406 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.220726 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.220730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.220731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.220924 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.221033 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.221343 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.221450 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:11:57.230746 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.219788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.14.209.189:34149)
0620 06:11:57.219891 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.219895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.221502 (+ 1607us) server_negotiation.cc:407] Connection header received
0620 06:11:57.221663 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.221666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.221707 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.221753 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.222708 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.223555 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.224815 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.225175 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.225246 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.227084 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.227089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.227092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.227124 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.228968 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.229491 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.229495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.229497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.229575 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.229857 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.229861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.229862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.230050 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.230160 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.230488 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.230601 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:11:57.230758 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.203223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.14.209.188:43821)
0620 06:11:57.203329 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.203334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.222304 (+ 18970us) server_negotiation.cc:407] Connection header received
0620 06:11:57.222458 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.222461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.222517 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.222565 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.223430 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.224179 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.225247 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.225626 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.225704 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.227308 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.227312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.227315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.227337 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.229099 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.229620 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.229624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.229625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.229702 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.229949 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.229952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.229953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.230125 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.230220 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.230507 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.230615 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:57.239240 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:57.220722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.14.209.188:43821)
0620 06:11:57.220829 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:57.220833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:57.231317 (+ 10484us) server_negotiation.cc:407] Connection header received
0620 06:11:57.231479 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:57.231482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:57.231523 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:57.231569 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:57.232429 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.233185 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.234394 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:57.234769 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:57.234870 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:57.236327 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:57.236331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:57.236333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:57.236357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:57.237780 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.238163 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.238167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.238168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.238246 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:57.238477 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:57.238480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:57.238482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:57.238672 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:57.238798 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:57.238959 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 06:11:57.239083 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:11:58.188553 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.187785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:47425)
0620 06:11:58.188088 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.188178 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
W20260620 06:11:58.188859 22252 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:46813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.189237 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.188724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:54707)
0620 06:11:58.189061 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.189140 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":65,"threads_started":1}
W20260620 06:11:58.189643 22389 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:46813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.189776 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.189323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:35473)
0620 06:11:58.189600 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.189678 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
W20260620 06:11:58.189993 22527 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:46813 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.700363 22595 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f5583e026484a63a4eaeaf5dc5afc4c)
I20260620 06:11:58.700600 22595 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:11:58.700911 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f5583e026484a63a4eaeaf5dc5afc4c)
I20260620 06:11:58.701109 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:11:58.701632 22595 leader_election.cc:288] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813), 138f268bef6d450e85d5b44f59531657 (127.14.209.188:43821)
I20260620 06:11:58.702193 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.701757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39460)
0620 06:11:58.701998 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.702080 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":56,"threads_started":1}
W20260620 06:11:58.702354 21874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.702672 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.702330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39468)
0620 06:11:58.702506 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.702577 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
W20260620 06:11:58.703001 22058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.703136 22596 leader_election.cc:288] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813), a9940460356c4d95a1242f7b59d82633 (127.14.209.189:34149)
I20260620 06:11:58.703410 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.702964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39482)
0620 06:11:58.703297 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.703347 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
W20260620 06:11:58.703547 21874 leader_election.cc:334] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.704269 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.704041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39494)
0620 06:11:58.704152 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.704207 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:11:58.704365 22058 leader_election.cc:334] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.706452 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.701831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:43821 (local address 127.0.0.1:60460)
0620 06:11:58.702136 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.702147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:58.702360 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:58.702894 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:58.702897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:58.702913 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:58.703028 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:58.703033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.704176 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.704180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:58.705365 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:58.705375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.706160 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.706163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:58.706228 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:58.706247 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:11:58.706310 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:11:58.706362 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":57,"threads_started":1}
I20260620 06:11:58.706532 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.702107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.14.209.188:43821)
0620 06:11:58.702321 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:58.702326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:58.702342 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:58.702523 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:58.702526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:58.702573 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:58.702630 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:58.703318 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.704089 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.705503 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.706030 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.706162 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:58.706209 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:58.706342 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:11:58.706420 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260620 06:11:58.706993 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9940460356c4d95a1242f7b59d82633" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "138f268bef6d450e85d5b44f59531657" is_pre_election: true
I20260620 06:11:58.707155 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.703096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.14.209.189:34149)
0620 06:11:58.703425 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:58.703435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:58.703450 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:58.703499 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:58.703502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:58.703559 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:58.703651 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:11:58.703929 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.704639 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.705931 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.706543 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.706675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:58.706720 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:58.706937 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 06:11:58.707008 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":84,"threads_started":1}
I20260620 06:11:58.707158 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.702739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:34149 (local address 127.0.0.1:43030)
0620 06:11:58.702814 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.702829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:58.702903 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:58.703706 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:58.703709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:58.703729 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:11:58.703804 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:58.703809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.704750 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.704753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:58.705800 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:58.705809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:58.706688 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:58.706692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:58.706787 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:58.706803 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:11:58.706919 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 06:11:58.706982 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"thread_start_us":55,"threads_started":1}
I20260620 06:11:58.707258 22081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9940460356c4d95a1242f7b59d82633 in term 1.
I20260620 06:11:58.707614 21875 leader_election.cc:302] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 138f268bef6d450e85d5b44f59531657, a9940460356c4d95a1242f7b59d82633; no voters: 9f5583e026484a63a4eaeaf5dc5afc4c
I20260620 06:11:58.707628 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "138f268bef6d450e85d5b44f59531657" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9940460356c4d95a1242f7b59d82633" is_pre_election: true
I20260620 06:11:58.707801 21898 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138f268bef6d450e85d5b44f59531657 in term 1.
I20260620 06:11:58.707890 22595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:11:58.707989 22595 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f5583e026484a63a4eaeaf5dc5afc4c)
I20260620 06:11:58.708061 22595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:11:58.708150 22059 leader_election.cc:302] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 138f268bef6d450e85d5b44f59531657, a9940460356c4d95a1242f7b59d82633; no voters: 9f5583e026484a63a4eaeaf5dc5afc4c
I20260620 06:11:58.708319 22596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:11:58.708364 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f5583e026484a63a4eaeaf5dc5afc4c)
I20260620 06:11:58.708401 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:11:58.709074 22595 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:11:58.709318 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:11:58.709441 22595 leader_election.cc:288] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 2 election: Requested vote from peers 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813), 138f268bef6d450e85d5b44f59531657 (127.14.209.188:43821)
I20260620 06:11:58.709652 22596 leader_election.cc:288] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [CANDIDATE]: Term 2 election: Requested vote from peers 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813), a9940460356c4d95a1242f7b59d82633 (127.14.209.189:34149)
I20260620 06:11:58.709791 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9940460356c4d95a1242f7b59d82633" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "138f268bef6d450e85d5b44f59531657"
I20260620 06:11:58.709971 22081 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a9940460356c4d95a1242f7b59d82633 in current term 2: Already voted for candidate 138f268bef6d450e85d5b44f59531657 in this term.
I20260620 06:11:58.710072 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "138f268bef6d450e85d5b44f59531657" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9940460356c4d95a1242f7b59d82633"
I20260620 06:11:58.710225 21898 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 138f268bef6d450e85d5b44f59531657 in current term 2: Already voted for candidate a9940460356c4d95a1242f7b59d82633 in this term.
I20260620 06:11:58.710364 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.709515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39506)
0620 06:11:58.709866 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.709938 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260620 06:11:58.710393 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.710046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39522)
0620 06:11:58.710280 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.710327 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 06:11:58.710968 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.710756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39548)
0620 06:11:58.710863 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.710914 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 06:11:58.711001 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:58.710749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:39532)
0620 06:11:58.710868 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:58.710950 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 06:11:58.711089 21874 leader_election.cc:334] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
W20260620 06:11:58.711114 22058 leader_election.cc:334] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:11:58.711145 21874 leader_election.cc:302] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9940460356c4d95a1242f7b59d82633; no voters: 138f268bef6d450e85d5b44f59531657, 9f5583e026484a63a4eaeaf5dc5afc4c
I20260620 06:11:58.711220 22058 leader_election.cc:302] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138f268bef6d450e85d5b44f59531657; no voters: 9f5583e026484a63a4eaeaf5dc5afc4c, a9940460356c4d95a1242f7b59d82633
I20260620 06:11:58.711295 22595 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 06:11:58.711488 22596 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 06:11:59.190060 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:59.189469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:46399)
0620 06:11:59.189864 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:59.189928 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":104,"threads_started":1}
I20260620 06:11:59.191262 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:59.190161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:55773)
0620 06:11:59.191106 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:59.191169 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":58,"threads_started":1}
I20260620 06:11:59.191308 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:59.190743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:33187)
0620 06:11:59.191170 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:59.191218 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":56,"threads_started":1}
I20260620 06:12:00.191241 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.190712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:48909)
0620 06:12:00.191028 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.191119 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":76,"threads_started":1}
I20260620 06:12:00.192996 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.192031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:37097)
0620 06:12:00.192801 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.192875 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":60,"threads_started":1}
I20260620 06:12:00.193682 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.192428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:46823)
0620 06:12:00.193528 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.193589 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":91,"threads_started":1}
I20260620 06:12:00.600593 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:00.600703 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:12:00.601135 22611 leader_election.cc:288] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813), 138f268bef6d450e85d5b44f59531657 (127.14.209.188:43821)
I20260620 06:12:00.601512 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9940460356c4d95a1242f7b59d82633" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "138f268bef6d450e85d5b44f59531657" is_pre_election: true
I20260620 06:12:00.601631 22081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9940460356c4d95a1242f7b59d82633 in term 2.
I20260620 06:12:00.601691 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.601271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:40798)
0620 06:12:00.601508 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.601556 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":52,"threads_started":1}
I20260620 06:12:00.601928 21875 leader_election.cc:302] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138f268bef6d450e85d5b44f59531657, a9940460356c4d95a1242f7b59d82633; no voters:
I20260620 06:12:00.602109 22611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 06:12:00.602167 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:00.602196 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:12:00.602573 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.602337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:40806)
0620 06:12:00.602482 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.602523 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 06:12:00.602676 21874 leader_election.cc:334] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:12:00.603430 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:12:00.603758 22611 leader_election.cc:288] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 3 election: Requested vote from peers 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813), 138f268bef6d450e85d5b44f59531657 (127.14.209.188:43821)
I20260620 06:12:00.604058 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.603864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:40814)
0620 06:12:00.603957 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.604008 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 06:12:00.604084 22081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9940460356c4d95a1242f7b59d82633" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "138f268bef6d450e85d5b44f59531657"
I20260620 06:12:00.604204 22081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:12:00.604692 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.604520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:40820)
0620 06:12:00.604606 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.604646 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:12:00.604854 21874 leader_election.cc:334] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
I20260620 06:12:00.605306 22081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9940460356c4d95a1242f7b59d82633 in term 3.
I20260620 06:12:00.605554 21875 leader_election.cc:302] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 138f268bef6d450e85d5b44f59531657, a9940460356c4d95a1242f7b59d82633; no voters: 9f5583e026484a63a4eaeaf5dc5afc4c
I20260620 06:12:00.605712 22611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:12:00.605872 22611 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [term 3 LEADER]: Becoming Leader. State: Replica: a9940460356c4d95a1242f7b59d82633, State: Running, Role: LEADER
I20260620 06:12:00.606129 22611 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } }
I20260620 06:12:00.607159 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9940460356c4d95a1242f7b59d82633. Latest consensus state: current_term: 3 leader_uuid: "a9940460356c4d95a1242f7b59d82633" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:12:00.607280 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:00.607559 22616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:00.609670 22616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:00.610076 22616 catalog_manager.cc:1259] Loaded cluster ID: e373f05ea2c3438e82bb64d3fba9d0ad
I20260620 06:12:00.610121 22616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:00.610759 22616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:00.611217 22616 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Loaded TSK: 0
I20260620 06:12:00.611871 22616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:12:00.687531 21887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:12:00.690884 21887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:00.693794 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.693523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:40844)
0620 06:12:00.693661 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.693719 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 06:12:00.693795 22081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [term 3 FOLLOWER]: Refusing update from remote peer a9940460356c4d95a1242f7b59d82633: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 06:12:00.694243 22611 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 06:12:00.694597 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.694330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.0.0.1:40858)
0620 06:12:00.694454 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.694511 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 06:12:00.694725 21874 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 -> Peer 9f5583e026484a63a4eaeaf5dc5afc4c (127.14.209.190:46813): Couldn't send request to peer 9f5583e026484a63a4eaeaf5dc5afc4c. Status: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 06:12:00.695745 22618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9940460356c4d95a1242f7b59d82633. Latest consensus state: current_term: 3 leader_uuid: "a9940460356c4d95a1242f7b59d82633" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:12:00.695885 22618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:12:00.696832 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a9940460356c4d95a1242f7b59d82633" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:12:00.696947 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633 [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:00.697275 22618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a9940460356c4d95a1242f7b59d82633" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9f5583e026484a63a4eaeaf5dc5afc4c" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46813 } } peers { permanent_uuid: "a9940460356c4d95a1242f7b59d82633" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 34149 } attrs { promote: false } } peers { permanent_uuid: "138f268bef6d450e85d5b44f59531657" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 43821 } attrs { promote: false } } }
I20260620 06:12:00.697408 22618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f268bef6d450e85d5b44f59531657 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:12:00.697571 22619 mvcc.cc:205] Tried to move back new op lower bound from 7298809531157397504 to 7298809530805010432. Current Snapshot: MvccSnapshot[applied={T|T < 7298809531157397504}]
I20260620 06:12:00.706723 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.702302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:42033 (local address 127.0.0.1:42562)
0620 06:12:00.702438 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.702452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:00.702541 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:00.703044 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:00.703047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:00.703102 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:00.703221 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:00.703231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.704350 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.704354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:00.705529 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:00.705538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.706342 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.706348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:00.706431 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.706455 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:12:00.706543 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:12:00.706606 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 06:12:00.706785 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.702410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.14.209.130:42033)
0620 06:12:00.702738 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:00.702744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:00.702758 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:12:00.702809 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:00.702818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:00.702882 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:00.703331 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:00.703444 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.704188 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.705660 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.706248 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.706359 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.706407 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:00.706568 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 06:12:00.706644 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":62,"threads_started":1}
I20260620 06:12:00.707198 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.702571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39157 (local address 127.0.0.1:55890)
0620 06:12:00.703177 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.703190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:00.703267 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:00.703746 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:00.703749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:00.703764 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:00.703836 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:00.703841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.704676 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.704681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:00.705450 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:00.705457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.706464 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.706476 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:00.706587 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.706610 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:12:00.706677 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:12:00.706740 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":25,"thread_start_us":114,"threads_started":1}
I20260620 06:12:00.707434 22322 tablet_service.cc:1467] Processing CreateTablet for tablet a226f15fc38140009b95766c070b0e9e (DEFAULT_TABLE table=test-table [id=0df2ad53ef3f4f54b0e815488b62dd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:00.707515 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.702865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:39817 (local address 127.0.0.1:49360)
0620 06:12:00.703440 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.703452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:00.703532 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:00.703917 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:00.703920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:00.703935 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:00.704024 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:00.704053 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.705222 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.705225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:00.706259 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:00.706268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.707168 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.707171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:00.707255 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.707279 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:12:00.707341 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:12:00.707401 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":81,"threads_started":1}
I20260620 06:12:00.707708 22322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a226f15fc38140009b95766c070b0e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:00.707742 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.702941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.14.209.129:39157)
0620 06:12:00.703453 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:00.703458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:00.703480 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:12:00.703525 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:00.703527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:00.703569 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:00.703647 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:00.703946 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.704570 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.705565 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.706238 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.706359 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.706407 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:00.707531 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0620 06:12:00.707608 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":62,"threads_started":1}
I20260620 06:12:00.707899 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.703199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.14.209.131:39817)
0620 06:12:00.703714 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:00.703718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:00.703728 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:12:00.703769 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:00.703771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:00.703804 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:00.703897 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:00.704144 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.705037 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.706388 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.707075 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.707182 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.707229 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:00.707669 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 06:12:00.707754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":75,"threads_started":1}
I20260620 06:12:00.708235 22185 tablet_service.cc:1467] Processing CreateTablet for tablet a226f15fc38140009b95766c070b0e9e (DEFAULT_TABLE table=test-table [id=0df2ad53ef3f4f54b0e815488b62dd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:00.708379 22460 tablet_service.cc:1467] Processing CreateTablet for tablet a226f15fc38140009b95766c070b0e9e (DEFAULT_TABLE table=test-table [id=0df2ad53ef3f4f54b0e815488b62dd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:00.708452 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a226f15fc38140009b95766c070b0e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:00.708590 22460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a226f15fc38140009b95766c070b0e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:00.710505 22629 tablet_bootstrap.cc:492] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Bootstrap starting.
I20260620 06:12:00.710871 22630 tablet_bootstrap.cc:492] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Bootstrap starting.
I20260620 06:12:00.711272 22631 tablet_bootstrap.cc:492] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51: Bootstrap starting.
I20260620 06:12:00.711519 22629 tablet_bootstrap.cc:654] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:00.711714 22630 tablet_bootstrap.cc:654] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:00.712282 22631 tablet_bootstrap.cc:654] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:00.712814 22629 tablet_bootstrap.cc:492] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: No bootstrap required, opened a new log
I20260620 06:12:00.712961 22629 ts_tablet_manager.cc:1397] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:00.713443 22629 raft_consensus.cc:348] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.713572 22629 raft_consensus.cc:374] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:00.713608 22629 raft_consensus.cc:729] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7af512cd94d7288ddfdf4a9f2fc8b, State: Initialized, Role: FOLLOWER
I20260620 06:12:00.713747 22631 tablet_bootstrap.cc:492] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51: No bootstrap required, opened a new log
I20260620 06:12:00.713738 22629 consensus_queue.cc:260] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.713827 22631 ts_tablet_manager.cc:1397] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:00.714068 22629 ts_tablet_manager.cc:1428] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 06:12:00.714268 22631 raft_consensus.cc:348] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.714433 22631 raft_consensus.cc:374] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:00.714491 22631 raft_consensus.cc:729] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d206ba20aa4807b8070b2ef009df51, State: Initialized, Role: FOLLOWER
I20260620 06:12:00.714643 22631 consensus_queue.cc:260] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.715034 22631 ts_tablet_manager.cc:1428] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:00.715304 22630 tablet_bootstrap.cc:492] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: No bootstrap required, opened a new log
I20260620 06:12:00.715394 22630 ts_tablet_manager.cc:1397] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 06:12:00.715829 22630 raft_consensus.cc:348] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.715931 22630 raft_consensus.cc:374] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:00.715965 22630 raft_consensus.cc:729] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee8328a367745b286e8981448426b46, State: Initialized, Role: FOLLOWER
I20260620 06:12:00.716086 22630 consensus_queue.cc:260] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.716398 22630 ts_tablet_manager.cc:1428] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:00.716588 22390 heartbeater.cc:503] Master 127.14.209.189:34149 was elected leader, sending a full tablet report...
I20260620 06:12:00.717036 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.716572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:44875)
0620 06:12:00.716889 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.716948 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":86,"threads_started":1}
I20260620 06:12:00.718165 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.717774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:34683)
0620 06:12:00.718012 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.718068 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 06:12:00.718803 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.714454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:54781)
0620 06:12:00.714789 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.718701 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260620 06:12:00.719038 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.718830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:55369)
0620 06:12:00.718931 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.718980 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 06:12:00.719287 22253 heartbeater.cc:503] Master 127.14.209.189:34149 was elected leader, sending a full tablet report...
I20260620 06:12:00.719696 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.719277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:44665)
0620 06:12:00.719552 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.719618 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260620 06:12:00.719973 22528 heartbeater.cc:503] Master 127.14.209.189:34149 was elected leader, sending a full tablet report...
I20260620 06:12:00.720577 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.720388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:34135)
0620 06:12:00.720473 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.720520 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 06:12:00.742722 22637 raft_consensus.cc:482] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:00.742843 22637 raft_consensus.cc:504] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.743203 22637 leader_election.cc:288] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033), aee8328a367745b286e8981448426b46 (127.14.209.131:39817)
I20260620 06:12:00.743611 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a226f15fc38140009b95766c070b0e9e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" is_pre_election: true
I20260620 06:12:00.743614 22480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a226f15fc38140009b95766c070b0e9e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee8328a367745b286e8981448426b46" is_pre_election: true
I20260620 06:12:00.743752 22342 raft_consensus.cc:2454] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 0.
I20260620 06:12:00.743763 22480 raft_consensus.cc:2454] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 0.
I20260620 06:12:00.744089 22138 leader_election.cc:302] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d206ba20aa4807b8070b2ef009df51, c7e7af512cd94d7288ddfdf4a9f2fc8b; no voters:
I20260620 06:12:00.744242 22637 raft_consensus.cc:2781] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:00.744298 22637 raft_consensus.cc:482] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:00.744340 22637 raft_consensus.cc:3037] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:00.745213 22637 raft_consensus.cc:504] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.745512 22637 leader_election.cc:288] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 election: Requested vote from peers c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033), aee8328a367745b286e8981448426b46 (127.14.209.131:39817)
I20260620 06:12:00.745786 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a226f15fc38140009b95766c070b0e9e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b"
I20260620 06:12:00.745893 22342 raft_consensus.cc:3037] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:00.745867 22480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a226f15fc38140009b95766c070b0e9e" candidate_uuid: "49d206ba20aa4807b8070b2ef009df51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee8328a367745b286e8981448426b46"
I20260620 06:12:00.745998 22480 raft_consensus.cc:3037] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:00.746707 22342 raft_consensus.cc:2454] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 1.
I20260620 06:12:00.746848 22480 raft_consensus.cc:2454] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49d206ba20aa4807b8070b2ef009df51 in term 1.
I20260620 06:12:00.746959 22138 leader_election.cc:302] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49d206ba20aa4807b8070b2ef009df51, c7e7af512cd94d7288ddfdf4a9f2fc8b; no voters:
I20260620 06:12:00.747153 22637 raft_consensus.cc:2781] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:00.747382 22637 raft_consensus.cc:686] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 1 LEADER]: Becoming Leader. State: Replica: 49d206ba20aa4807b8070b2ef009df51, State: Running, Role: LEADER
I20260620 06:12:00.747550 22637 consensus_queue.cc:237] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } }
I20260620 06:12:00.748108 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.747806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:57059)
0620 06:12:00.747999 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.748042 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 06:12:00.749166 21888 catalog_manager.cc:5495] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129). New cstate: current_term: 1 leader_uuid: "49d206ba20aa4807b8070b2ef009df51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49d206ba20aa4807b8070b2ef009df51" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20260620 06:12:00.770320 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.767014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.14.209.129:39157)
0620 06:12:00.767170 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:00.767175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:00.767198 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:12:00.767376 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:00.767378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:00.767421 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:00.767492 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:00.768020 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.768531 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.769451 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:00.769660 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:00.769738 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:00.769951 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:00.770016 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:00.770071 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:00.770187 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 06:12:00.770228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 06:12:00.772332 22480 raft_consensus.cc:1261] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Refusing update from remote peer 49d206ba20aa4807b8070b2ef009df51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:00.772332 22342 raft_consensus.cc:1261] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Refusing update from remote peer 49d206ba20aa4807b8070b2ef009df51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:00.772745 22641 consensus_queue.cc:1035] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e7af512cd94d7288ddfdf4a9f2fc8b" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:00.772895 22637 consensus_queue.cc:1035] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee8328a367745b286e8981448426b46" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:00.775617 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.775336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:46491)
0620 06:12:00.775491 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.775539 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 06:12:00.775859 22529 tablet.cc:2369] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:12:00.775290 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.774716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:47537)
0620 06:12:00.775102 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.775193 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 06:12:00.778009 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.777759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:41499)
0620 06:12:00.777894 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.777945 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 06:12:00.778029 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.777740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.130:54991)
0620 06:12:00.777916 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.777966 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 06:12:00.778324 22644 mvcc.cc:205] Tried to move back new op lower bound from 7298809531479367680 to 7298809531383353344. Current Snapshot: MvccSnapshot[applied={T|T < 7298809531479367680}]
I20260620 06:12:00.778672 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.778438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.129:34831)
0620 06:12:00.778518 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.778564 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 06:12:00.779342 22645 mvcc.cc:205] Tried to move back new op lower bound from 7298809531479367680 to 7298809531383353344. Current Snapshot: MvccSnapshot[applied={T|T < 7298809531479367680}]
I20260620 06:12:00.780068 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.779839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46813 (local address 127.14.209.131:44371)
0620 06:12:00.779964 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.780011 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:46813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 06:12:00.783294 22646 mvcc.cc:205] Tried to move back new op lower bound from 7298809531479367680 to 7298809531383353344. Current Snapshot: MvccSnapshot[applied={T|T < 7298809531479367680}]
I20260620 06:12:00.785424 21888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:00.785575 21888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:00.789027 21888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9940460356c4d95a1242f7b59d82633: Sending DeleteTablet for 3 replicas of tablet a226f15fc38140009b95766c070b0e9e
I20260620 06:12:00.789742 22460 tablet_service.cc:1514] Processing DeleteTablet for tablet a226f15fc38140009b95766c070b0e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:00 UTC) from {username='oryx', principal='oryx/127.14.209.189@KRBTEST.COM'} at 127.0.0.1:49360
I20260620 06:12:00.789749 22322 tablet_service.cc:1514] Processing DeleteTablet for tablet a226f15fc38140009b95766c070b0e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:00 UTC) from {username='oryx', principal='oryx/127.14.209.189@KRBTEST.COM'} at 127.0.0.1:42562
I20260620 06:12:00.789849 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet a226f15fc38140009b95766c070b0e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:00 UTC) from {username='oryx', principal='oryx/127.14.209.189@KRBTEST.COM'} at 127.0.0.1:55890
I20260620 06:12:00.790067 22654 tablet_replica.cc:330] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: stopping tablet replica
I20260620 06:12:00.790114 22655 tablet_replica.cc:330] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51: stopping tablet replica
I20260620 06:12:00.790194 22655 raft_consensus.cc:2229] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:12:00.790184 22654 raft_consensus.cc:2229] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:00.790288 22654 raft_consensus.cc:2258] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:00.790324 22655 raft_consensus.cc:2258] T a226f15fc38140009b95766c070b0e9e P 49d206ba20aa4807b8070b2ef009df51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:00.790537 22653 tablet_replica.cc:330] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: stopping tablet replica
I20260620 06:12:00.790664 22653 raft_consensus.cc:2229] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:00.790805 22653 raft_consensus.cc:2258] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:00.790808 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 22032
I20260620 06:12:00.791091 22654 ts_tablet_manager.cc:1905] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:00.793370 22653 ts_tablet_manager.cc:1905] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:00.793511 22654 ts_tablet_manager.cc:1918] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:00.793617 22654 log.cc:1196] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/a226f15fc38140009b95766c070b0e9e
I20260620 06:12:00.793871 22654 ts_tablet_manager.cc:1939] T a226f15fc38140009b95766c070b0e9e P aee8328a367745b286e8981448426b46: Deleting consensus metadata
I20260620 06:12:00.794450 21875 catalog_manager.cc:4841] TS aee8328a367745b286e8981448426b46 (127.14.209.131:39817): tablet a226f15fc38140009b95766c070b0e9e (table test-table [id=0df2ad53ef3f4f54b0e815488b62dd41]) successfully deleted
I20260620 06:12:00.796468 22653 ts_tablet_manager.cc:1918] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:00.796555 22653 log.cc:1196] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/a226f15fc38140009b95766c070b0e9e
I20260620 06:12:00.796826 22653 ts_tablet_manager.cc:1939] T a226f15fc38140009b95766c070b0e9e P c7e7af512cd94d7288ddfdf4a9f2fc8b: Deleting consensus metadata
I20260620 06:12:00.797511 21872 catalog_manager.cc:4841] TS c7e7af512cd94d7288ddfdf4a9f2fc8b (127.14.209.130:42033): tablet a226f15fc38140009b95766c070b0e9e (table test-table [id=0df2ad53ef3f4f54b0e815488b62dd41]) successfully deleted
I20260620 06:12:00.799266 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 22259
I20260620 06:12:00.799494 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:00.799208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39157 (local address 127.0.0.1:55918)
0620 06:12:00.799342 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:00.799405 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:12:00.800814 21872 catalog_manager.cc:4584] TS 49d206ba20aa4807b8070b2ef009df51 (127.14.209.129:39157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a226f15fc38140009b95766c070b0e9e: Network error: Client connection negotiation failed: client connection to 127.14.209.129:39157: connect: Connection refused (error 111)
I20260620 06:12:00.805661 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 22395
I20260620 06:12:00.811430 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21752
I20260620 06:12:00.817343 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 21854
2026-06-20T06:12:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7054 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:12:00 dist-test-slave-zt01 krb5kdc[22658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:12:00 dist-test-slave-zt01 krb5kdc[22658](info): set up 2 sockets
Jun 20 06:12:00 dist-test-slave-zt01 krb5kdc[22658](info): commencing operation
krb5kdc: starting...
W20260620 06:12:01.867627 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:12:01 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab.
Entry for principal oryx/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab.
I20260620 06:12:01.983023 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45585
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab
--principal=oryx/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:45585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:02.087870 22671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:02.088102 22671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:02.088140 22671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:02.091946 22671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:02.091986 22671 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:12:02.092003 22671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:02.092021 22671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:02.092037 22671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:02.092052 22671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:02.096956 22671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:45585
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.190.keytab
--principal=oryx/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:02.097793 22671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:02.098639 22671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:02.108415 22679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.108465 22677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.108582 22676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:02.109089 22671 server_base.cc:1034] running on GCE node
I20260620 06:12:02.109586 22671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:02.110507 22671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:02.111687 22671 hybrid_clock.cc:648] HybridClock initialized: now 1781935922111664 us; error 39 us; skew 500 ppm
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:02.114319 22671 init.cc:377] Logged in from keytab as oryx/127.14.209.190@KRBTEST.COM (short username oryx)
I20260620 06:12:02.115219 22671 webserver.cc:460] Webserver started at http://127.14.209.190:42297/ using document root <none> and password file <none>
I20260620 06:12:02.115729 22671 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:02.115814 22671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:02.116034 22671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:02.118008 22671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "913c2dd4aaa04b629070fb4ac5e96a0b"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "9b77c2c861ba5ee56448864725e0c2ad"
server_key_iv: "85f67ea1faa11d85a999eb2183313930"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.118535 22671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "913c2dd4aaa04b629070fb4ac5e96a0b"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "9b77c2c861ba5ee56448864725e0c2ad"
server_key_iv: "85f67ea1faa11d85a999eb2183313930"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.121927 22671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:12:02.124459 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.126199 22671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:12:02.126442 22671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "913c2dd4aaa04b629070fb4ac5e96a0b"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "9b77c2c861ba5ee56448864725e0c2ad"
server_key_iv: "85f67ea1faa11d85a999eb2183313930"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.126564 22671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:02.136649 22671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:02.139770 22671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:02.139971 22671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:02.147897 22671 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:45585
I20260620 06:12:02.147912 22748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:45585 every 8 connection(s)
I20260620 06:12:02.148768 22671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:12:02.151391 22749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:02.156761 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Bootstrap starting.
I20260620 06:12:02.158471 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 22671
I20260620 06:12:02.158603 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:12:02.158900 15174 external_mini_cluster.cc:1428] Setting key b15de8e24b9074cf4e62ac6d0fcae887
I20260620 06:12:02.159322 22749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:02.160130 22749 log.cc:824] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:02.161836 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: No bootstrap required, opened a new log
I20260620 06:12:02.165120 22749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45585 } }
I20260620 06:12:02.165391 22749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:02.165470 22749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913c2dd4aaa04b629070fb4ac5e96a0b, State: Initialized, Role: FOLLOWER
I20260620 06:12:02.165905 22749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45585 } }
I20260620 06:12:02.166057 22749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:12:02.166145 22749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:12:02.166236 22749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:02.167032 22749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45585 } }
I20260620 06:12:02.167405 22749 leader_election.cc:302] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913c2dd4aaa04b629070fb4ac5e96a0b; no voters:
I20260620 06:12:02.167740 22749 leader_election.cc:288] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:02.167845 22754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:02.168105 22754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [term 1 LEADER]: Becoming Leader. State: Replica: 913c2dd4aaa04b629070fb4ac5e96a0b, State: Running, Role: LEADER
I20260620 06:12:02.168431 22754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45585 } }
I20260620 06:12:02.169000 22749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:12:02.170006 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913c2dd4aaa04b629070fb4ac5e96a0b. Latest consensus state: current_term: 1 leader_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45585 } } }
I20260620 06:12:02.170131 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:02.170432 22755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913c2dd4aaa04b629070fb4ac5e96a0b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45585 } } }
I20260620 06:12:02.170491 22763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:02.170548 22755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:02.173574 22763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:02.174221 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.160102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.14.209.190:45585)
0620 06:12:02.160515 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.160529 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.160562 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:12:02.161173 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.161188 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.161482 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.161796 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.162579 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.163877 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.165017 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.165640 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.165810 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.168522 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.168542 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.168557 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.168586 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.170698 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.171237 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.171242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.171248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.171373 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.171667 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.171672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.171674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.172150 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.172344 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.173052 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.173324 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":111,"threads_started":1}
I20260620 06:12:02.179774 22763 catalog_manager.cc:1347] Generated new cluster ID: cd80765b453d44e2b1e69d11d2bb1336
I20260620 06:12:02.179840 22763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:02.190784 22763 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:12:02.191769 22763 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:02.198088 22763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Generated new TSK 0
I20260620 06:12:02.198668 22763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab.
Entry for principal oryx/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab.
I20260620 06:12:02.230641 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45585
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:02.341696 22775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:02.341913 22775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:02.341953 22775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:02.346251 22775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:02.346306 22775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:02.346344 22775 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:12:02.346376 22775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:12:02.351207 22775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:45585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:02.352105 22775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:02.352941 22775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:02.362196 22780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.362221 22783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:02.362468 22775 server_base.cc:1034] running on GCE node
W20260620 06:12:02.362202 22781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:02.363518 22775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:02.364091 22775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:02.365285 22775 hybrid_clock.cc:648] HybridClock initialized: now 1781935922365259 us; error 49 us; skew 500 ppm
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:02.368012 22775 init.cc:377] Logged in from keytab as oryx/127.14.209.129@KRBTEST.COM (short username oryx)
I20260620 06:12:02.368798 22775 webserver.cc:460] Webserver started at http://127.14.209.129:36879/ using document root <none> and password file <none>
I20260620 06:12:02.369396 22775 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:02.369486 22775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:02.369684 22775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:02.371685 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "607a2035cdf140a9bbee73ce58d85868"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "b3f92b14da5b231dd66277a42fa5493b"
server_key_iv: "ad4e5e5e29f1275ae4d12a32468c5816"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.372213 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "607a2035cdf140a9bbee73ce58d85868"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "b3f92b14da5b231dd66277a42fa5493b"
server_key_iv: "ad4e5e5e29f1275ae4d12a32468c5816"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.375741 22775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:12:02.378080 22790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.379377 22775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:12:02.379518 22775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "607a2035cdf140a9bbee73ce58d85868"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "b3f92b14da5b231dd66277a42fa5493b"
server_key_iv: "ad4e5e5e29f1275ae4d12a32468c5816"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.379621 22775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:02.414927 22775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:02.418376 22775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:02.418629 22775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:02.420295 22775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:02.420377 22775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.420449 22775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:02.420492 22775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:02.431916 22775 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:37509
I20260620 06:12:02.431998 22905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:37509 every 8 connection(s)
I20260620 06:12:02.432792 22775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:12:02.434723 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.421591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:37049 (local address 127.14.209.190:45585)
0620 06:12:02.421806 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.421812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.422509 (+ 697us) server_negotiation.cc:407] Connection header received
0620 06:12:02.423469 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.423473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.423532 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.423640 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.425168 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.426045 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.427658 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.427893 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.428018 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.431216 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.431220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.431223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.431255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.432779 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.433291 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.433294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.433296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.433384 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.433798 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.433801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.433802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.433977 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.434054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.434508 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.434608 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:12:02.435482 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.421862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.14.209.129:37049)
0620 06:12:02.422309 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.422357 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.423258 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.423814 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.423823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.424341 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:02.425006 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.425046 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.426154 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.426158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.427533 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.427542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.428061 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.428065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.428267 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.428899 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:02.428917 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:02.430969 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:02.432903 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.432908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.432921 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.433167 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.433495 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.433498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.433500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.433664 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.434077 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:02.434082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:02.434281 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.434427 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.434641 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":130,"threads_started":1}
I20260620 06:12:02.436980 22906 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45585
I20260620 06:12:02.437137 22906 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:02.437553 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 22775
I20260620 06:12:02.437599 22906 heartbeater.cc:511] Master 127.14.209.190:45585 requested a full tablet report, sending...
I20260620 06:12:02.437688 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:12:02.438066 15174 external_mini_cluster.cc:1428] Setting key 99d3013ef0710937fc485d8e058f6311
I20260620 06:12:02.439157 22702 ts_manager.cc:194] Registered new tserver with Master: 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509)
I20260620 06:12:02.440673 22702 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:37049
I20260620 06:12:02.446846 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.439214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.14.209.129:43137)
0620 06:12:02.439551 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.439562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.439681 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.439908 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.439910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.440099 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:02.440183 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.440189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.441074 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.441079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.441925 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.441934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.442575 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.442578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.442665 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.443207 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:02.443218 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:02.443506 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:02.445491 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.445493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.445494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.445674 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.445975 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.445977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.445978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.446052 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.446467 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:02.446469 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:02.446538 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.446620 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.446714 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
I20260620 06:12:02.447000 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.439303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:43137 (local address 127.14.209.190:45585)
0620 06:12:02.439411 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.439417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.439633 (+ 216us) server_negotiation.cc:407] Connection header received
0620 06:12:02.439765 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.439768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.439812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.439900 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.440296 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.440931 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.442056 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.442447 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.442564 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.443645 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.443652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.443658 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.443687 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.445372 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.445778 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.445784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.445787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.445874 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.446145 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.446151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.446153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.446367 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.446473 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.446707 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.446828 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.14.209.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab.
Entry for principal oryx/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab.
I20260620 06:12:02.466154 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45585
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:02.571574 22915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:02.571784 22915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:02.571825 22915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:02.575663 22915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:02.575713 22915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:02.575752 22915 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:12:02.575783 22915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:12:02.580586 22915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:45585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:02.581382 22915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:02.582258 22915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:02.591212 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.591251 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.591276 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:02.591399 22915 server_base.cc:1034] running on GCE node
I20260620 06:12:02.591846 22915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:02.592414 22915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:02.593580 22915 hybrid_clock.cc:648] HybridClock initialized: now 1781935922593553 us; error 42 us; skew 500 ppm
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:02.596267 22915 init.cc:377] Logged in from keytab as oryx/127.14.209.130@KRBTEST.COM (short username oryx)
I20260620 06:12:02.597061 22915 webserver.cc:460] Webserver started at http://127.14.209.130:33477/ using document root <none> and password file <none>
I20260620 06:12:02.597569 22915 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:02.597628 22915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:02.597851 22915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:02.599773 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "56eb6c2999b44a76a4a219492b2425c2"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "bda0371dc8203b1062c192d5068d3283"
server_key_iv: "47881628de5fc7f469a533b23414e567"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.600302 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "56eb6c2999b44a76a4a219492b2425c2"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "bda0371dc8203b1062c192d5068d3283"
server_key_iv: "47881628de5fc7f469a533b23414e567"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.603901 22915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:12:02.606484 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.607676 22915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:12:02.607805 22915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "56eb6c2999b44a76a4a219492b2425c2"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "bda0371dc8203b1062c192d5068d3283"
server_key_iv: "47881628de5fc7f469a533b23414e567"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.607909 22915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:02.634963 22915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:02.638239 22915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:02.638432 22915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:02.639915 22915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:02.639986 22915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.640053 22915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:02.640090 22915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:02.650625 22915 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:41159
I20260620 06:12:02.651024 23045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:41159 every 8 connection(s)
I20260620 06:12:02.651583 22915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:12:02.653649 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.640674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:44093 (local address 127.14.209.190:45585)
0620 06:12:02.640796 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.640799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.641866 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 06:12:02.642986 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.642991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.643075 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.643173 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.644325 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.644848 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.646190 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.646515 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.646625 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.650086 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.650090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.650093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.650118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.651803 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.652381 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.652385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.652386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.652506 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.652806 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.652808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.652810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.652967 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.653044 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.653432 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.653537 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:12:02.654386 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.641191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.14.209.130:44093)
0620 06:12:02.641702 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.641751 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.642742 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.643419 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.643429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.643756 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:02.644174 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.644186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.644981 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.644985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.646063 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.646074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.646670 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.646674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.646844 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.647439 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:02.647462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:02.649928 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:02.651929 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.651936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.651950 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.652261 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.652601 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.652604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.652605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.652721 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.653055 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:02.653059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:02.653246 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.653387 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.653601 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
I20260620 06:12:02.655473 23046 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45585
I20260620 06:12:02.655622 23046 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:02.656064 23046 heartbeater.cc:511] Master 127.14.209.190:45585 requested a full tablet report, sending...
I20260620 06:12:02.657063 22703 ts_manager.cc:194] Registered new tserver with Master: 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159)
I20260620 06:12:02.657862 22703 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:44093
I20260620 06:12:02.661660 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 22915
I20260620 06:12:02.661813 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:12:02.662211 15174 external_mini_cluster.cc:1428] Setting key 978a1d37e20a113a48ebb8ff2ca718a9
I20260620 06:12:02.664661 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.657637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:59045 (local address 127.14.209.190:45585)
0620 06:12:02.657750 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.657753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.657794 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:12:02.657973 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.657978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.658040 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.658104 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.658597 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.659345 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.660153 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.660364 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.660457 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.661438 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.661442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.661444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.661470 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.663010 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.663480 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.663492 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.663493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.663580 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.663901 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.663903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.663904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.664092 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.664168 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.664472 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.664561 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:12:02.664865 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.657452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.14.209.130:59045)
0620 06:12:02.657730 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.657743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.657838 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.658142 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.658146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.658340 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:02.658458 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.658465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.659476 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.659479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.660036 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.660043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.660470 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.660472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.660538 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.661008 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:02.661019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:02.661290 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:02.663150 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.663152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.663153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.663371 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.663705 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.663708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.663711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.663805 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.664195 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:02.664200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:02.664303 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.664433 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.664590 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab.
Entry for principal oryx/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab.
I20260620 06:12:02.691257 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45585
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:02.795677 23055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:02.795909 23055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:02.795981 23055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:02.799906 23055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:02.799986 23055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:02.800052 23055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:12:02.800128 23055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:12:02.805063 23055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.209.190:45585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:02.805965 23055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:02.806874 23055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:02.816084 23061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.816107 23063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:02.816110 23060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:02.816447 23055 server_base.cc:1034] running on GCE node
I20260620 06:12:02.816844 23055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:02.817461 23055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:02.818666 23055 hybrid_clock.cc:648] HybridClock initialized: now 1781935922818653 us; error 26 us; skew 500 ppm
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:02.821584 23055 init.cc:377] Logged in from keytab as oryx/127.14.209.131@KRBTEST.COM (short username oryx)
I20260620 06:12:02.822372 23055 webserver.cc:460] Webserver started at http://127.14.209.131:45589/ using document root <none> and password file <none>
I20260620 06:12:02.822950 23055 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:02.823014 23055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:02.823277 23055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:02.825176 23055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "c06c396ac802414190479a82924c56e9"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "257e6ffb94002bce818d69949664fbe1"
server_key_iv: "2b790e82783b5af205dde66f4003b77a"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.825714 23055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "c06c396ac802414190479a82924c56e9"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "257e6ffb94002bce818d69949664fbe1"
server_key_iv: "2b790e82783b5af205dde66f4003b77a"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.829253 23055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:12:02.831709 23070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.832871 23055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:12:02.832962 23055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "c06c396ac802414190479a82924c56e9"
format_stamp: "Formatted at 2026-06-20 06:12:02 on dist-test-slave-zt01"
server_key: "257e6ffb94002bce818d69949664fbe1"
server_key_iv: "2b790e82783b5af205dde66f4003b77a"
server_key_version: "encryptionkey@0"
I20260620 06:12:02.833088 23055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:02.843194 23055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:02.845732 23055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:02.846022 23055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:02.847458 23055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:02.847512 23055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:02.847548 23055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:02.847565 23055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:02.858736 23055 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:45445
I20260620 06:12:02.858796 23185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:45445 every 8 connection(s)
I20260620 06:12:02.859652 23055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:12:02.863440 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.848815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:59877 (local address 127.14.209.190:45585)
0620 06:12:02.849029 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.849035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.850489 (+ 1454us) server_negotiation.cc:407] Connection header received
0620 06:12:02.852224 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.852229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.852303 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.852418 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.853820 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.854390 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.855852 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.856193 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.856314 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.859341 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.859349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.859353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.859402 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.861390 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.861901 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.861905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.861907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.861999 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.862341 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.862345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.862347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.862528 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.862631 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.863140 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.863258 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 06:12:02.864143 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.849507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.14.209.131:59877)
0620 06:12:02.850150 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.850314 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.851887 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.852650 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.852663 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.853115 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:02.853662 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.853676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.854534 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.854537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.855702 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.855713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.856357 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.856362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.856589 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.857268 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:02.857288 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:02.859094 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:02.861516 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.861522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.861535 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.861787 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.862100 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.862103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.862105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.862239 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.862641 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:02.862648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:02.862868 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.863042 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.863359 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":205,"threads_started":1}
I20260620 06:12:02.865314 23186 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45585
I20260620 06:12:02.865440 23186 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:02.865943 23186 heartbeater.cc:511] Master 127.14.209.190:45585 requested a full tablet report, sending...
I20260620 06:12:02.866315 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23055
I20260620 06:12:02.866488 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:12:02.866815 15174 external_mini_cluster.cc:1428] Setting key 0f5445d1be2a01e4aba743bebc4ed1cb
I20260620 06:12:02.867029 22702 ts_manager.cc:194] Registered new tserver with Master: c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
I20260620 06:12:02.867801 22702 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:59877
I20260620 06:12:02.869448 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:12:02.875073 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.867573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.14.209.131:34205)
0620 06:12:02.867870 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.867885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.867975 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.868380 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.868383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.868589 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:02.868710 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.868717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.869626 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.869629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.870261 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.870269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.870927 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.870931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.871009 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.871573 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:02.871584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:02.871884 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:02.873607 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.873610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.873612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.873799 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.874077 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:02.874081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:02.874083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:02.874168 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:02.874524 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:02.874527 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:02.874612 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.874720 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.874849 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260620 06:12:02.875041 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.867754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:34205 (local address 127.14.209.190:45585)
0620 06:12:02.867860 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.867866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.867939 (+ 73us) server_negotiation.cc:407] Connection header received
0620 06:12:02.868159 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.868161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.868207 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.868278 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.868821 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.869510 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.870383 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.870812 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.870926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.872033 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.872037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.872039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.872061 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.873477 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.873903 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.873906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.873908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.873988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.874264 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.874267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.874269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.874426 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.874532 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.874782 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.874895 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:12:02.892268 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:02.886719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.14.209.190:45585)
0620 06:12:02.886826 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.886829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.886840 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:12:02.886867 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.886869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.886898 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.886959 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.887668 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.888158 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.888992 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.889187 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.889261 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.892159 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41230: BlockingRecv error: recv got EOF from 127.0.0.1:41230 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 06:12:02 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:02.901316 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.893573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.14.209.190:45585)
0620 06:12:02.893700 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.893704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.893848 (+ 144us) server_negotiation.cc:407] Connection header received
0620 06:12:02.893982 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.893984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.894016 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.894095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:02.894703 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.895287 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.896047 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.896264 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.896364 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.898433 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:02.898436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:02.898437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:02.898457 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:02.899894 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.900271 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.900273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.900275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.900340 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:02.900646 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:02.900649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:02.900650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:02.900799 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:02.900894 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.901143 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.901222 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:12:02.904232 22703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:12:02.907171 22703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:02.921175 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.915776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:41159 (local address 127.0.0.1:34724)
0620 06:12:02.916264 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.916278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.916362 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.917338 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.917341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.917357 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:02.917425 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.917430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.918708 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.918710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.919560 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.919569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.920626 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.920629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.920744 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.920783 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.920902 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.921047 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":98,"threads_started":1}
I20260620 06:12:02.921186 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.916187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.14.209.130:41159)
0620 06:12:02.916776 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.916789 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.916811 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:12:02.916944 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.916952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.917203 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.917347 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:02.917526 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.918583 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.919680 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.920513 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.920654 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.920804 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.920938 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.921044 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":115,"threads_started":1}
I20260620 06:12:02.922055 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.915303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45445 (local address 127.0.0.1:58062)
0620 06:12:02.915895 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.915913 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.916003 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.917098 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.917101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.917117 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:02.917221 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.917226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.918405 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.918409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.919481 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.919497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.921687 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.921690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.921785 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.921812 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.921863 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.921930 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":91,"threads_started":1}
I20260620 06:12:02.922233 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.915040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.0.0.1:35292)
0620 06:12:02.915524 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.915568 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.915682 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.916719 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.916725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.916748 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:02.916889 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.916896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.918463 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.918466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.920608 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.920617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.921820 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.921823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.921917 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.921942 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.921994 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.922054 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":961,"thread_start_us":65,"threads_started":1}
I20260620 06:12:02.922583 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.915240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.14.209.129:37509)
0620 06:12:02.916200 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.916216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.916240 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:12:02.916322 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.916329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.916538 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.917039 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:02.917163 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.918324 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.920724 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.921690 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.922166 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.922276 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.922320 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.922401 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":55,"threads_started":1}
I20260620 06:12:02.922823 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.915821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.14.209.131:45445)
0620 06:12:02.916617 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.916630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.916655 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:12:02.916733 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.916739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.916929 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.917386 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:02.917518 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.918290 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.920773 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.921566 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.922372 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.922525 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.922571 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.922650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":106,"threads_started":1}
I20260620 06:12:02.924568 22980 tablet_service.cc:1467] Processing CreateTablet for tablet d7947d66c6a147ad98386f6bb90a2652 (DEFAULT_TABLE table=test-table [id=4c85aebd3cf3492da4bafd3e549299cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:02.924808 22840 tablet_service.cc:1467] Processing CreateTablet for tablet d7947d66c6a147ad98386f6bb90a2652 (DEFAULT_TABLE table=test-table [id=4c85aebd3cf3492da4bafd3e549299cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:02.925097 23120 tablet_service.cc:1467] Processing CreateTablet for tablet d7947d66c6a147ad98386f6bb90a2652 (DEFAULT_TABLE table=test-table [id=4c85aebd3cf3492da4bafd3e549299cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:02.925560 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7947d66c6a147ad98386f6bb90a2652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:02.925597 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7947d66c6a147ad98386f6bb90a2652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:02.925907 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7947d66c6a147ad98386f6bb90a2652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:02.932613 23214 tablet_bootstrap.cc:492] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Bootstrap starting.
I20260620 06:12:02.933209 23216 tablet_bootstrap.cc:492] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Bootstrap starting.
I20260620 06:12:02.933531 23215 tablet_bootstrap.cc:492] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Bootstrap starting.
I20260620 06:12:02.934577 23214 tablet_bootstrap.cc:654] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:02.935153 23215 tablet_bootstrap.cc:654] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:02.935355 23214 log.cc:824] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:02.935832 23215 log.cc:824] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:02.937307 23214 tablet_bootstrap.cc:492] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: No bootstrap required, opened a new log
I20260620 06:12:02.937551 23214 ts_tablet_manager.cc:1397] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:12:02.937557 23216 tablet_bootstrap.cc:654] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:02.937913 23215 tablet_bootstrap.cc:492] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: No bootstrap required, opened a new log
I20260620 06:12:02.938133 23215 ts_tablet_manager.cc:1397] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 06:12:02.938344 23216 log.cc:824] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:02.939774 23216 tablet_bootstrap.cc:492] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: No bootstrap required, opened a new log
I20260620 06:12:02.939962 23216 ts_tablet_manager.cc:1397] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 06:12:02.941385 23214 raft_consensus.cc:348] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.941636 23214 raft_consensus.cc:374] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:02.941740 23214 raft_consensus.cc:729] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607a2035cdf140a9bbee73ce58d85868, State: Initialized, Role: FOLLOWER
I20260620 06:12:02.941768 23215 raft_consensus.cc:348] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.942071 23215 raft_consensus.cc:374] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:02.942133 23215 raft_consensus.cc:729] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Initialized, Role: FOLLOWER
I20260620 06:12:02.942196 23214 consensus_queue.cc:260] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.942745 22906 heartbeater.cc:503] Master 127.14.209.190:45585 was elected leader, sending a full tablet report...
I20260620 06:12:02.943125 23214 ts_tablet_manager.cc:1428] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:12:02.943158 23216 raft_consensus.cc:348] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.943405 23216 raft_consensus.cc:374] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:02.943463 23216 raft_consensus.cc:729] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06c396ac802414190479a82924c56e9, State: Initialized, Role: FOLLOWER
I20260620 06:12:02.942587 23215 consensus_queue.cc:260] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.943828 23216 consensus_queue.cc:260] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.944236 23046 heartbeater.cc:503] Master 127.14.209.190:45585 was elected leader, sending a full tablet report...
I20260620 06:12:02.944530 23215 ts_tablet_manager.cc:1428] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:12:02.945387 23186 heartbeater.cc:503] Master 127.14.209.190:45585 was elected leader, sending a full tablet report...
I20260620 06:12:02.945626 23216 ts_tablet_manager.cc:1428] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:12:02.974115 23222 raft_consensus.cc:482] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:02.974291 23222 raft_consensus.cc:504] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.975371 23222 leader_election.cc:288] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159)
I20260620 06:12:02.979631 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.975662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.14.209.131:32915)
0620 06:12:02.975791 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.975808 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.975888 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.976205 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.976208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.976224 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:02.976355 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.976362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.977380 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.977383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.978660 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.978670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.979308 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.979311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.979393 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.979411 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.979469 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.979524 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260620 06:12:02.979681 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.975706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:32915 (local address 127.14.209.129:37509)
0620 06:12:02.975829 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.975833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.975846 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:02.976050 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.976053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.976100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.976184 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:02.976491 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.977253 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.978783 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.979183 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.979315 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.979370 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.979506 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.979573 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:12:02.980330 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7947d66c6a147ad98386f6bb90a2652" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868" is_pre_election: true
I20260620 06:12:02.980412 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.975921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:41159 (local address 127.14.209.131:41153)
0620 06:12:02.976224 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:02.976236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:02.976358 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:02.976621 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:02.976624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:02.976640 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:02.976782 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.976788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.977659 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.977662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.979147 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:02.979155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.980012 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.980015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:02.980115 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.980131 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:12:02.980219 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:12:02.980281 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":121,"threads_started":1}
I20260620 06:12:02.980397 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.976090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:41153 (local address 127.14.209.130:41159)
0620 06:12:02.976207 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.976211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.976303 (+ 92us) server_negotiation.cc:407] Connection header received
0620 06:12:02.976452 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.976458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.976512 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.976608 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:02.976905 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.977550 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.979271 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.979875 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.979999 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.980044 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.980233 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.980298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:12:02.980602 22860 raft_consensus.cc:2454] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 0.
I20260620 06:12:02.981072 23071 leader_election.cc:302] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607a2035cdf140a9bbee73ce58d85868, c06c396ac802414190479a82924c56e9; no voters:
I20260620 06:12:02.981159 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7947d66c6a147ad98386f6bb90a2652" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eb6c2999b44a76a4a219492b2425c2" is_pre_election: true
I20260620 06:12:02.981298 23222 raft_consensus.cc:2781] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:02.981381 23222 raft_consensus.cc:482] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:02.981429 23222 raft_consensus.cc:3037] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:02.981400 23000 raft_consensus.cc:2454] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 0.
I20260620 06:12:02.982486 23222 raft_consensus.cc:504] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.982741 23222 leader_election.cc:288] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 election: Requested vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159)
I20260620 06:12:02.983104 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7947d66c6a147ad98386f6bb90a2652" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868"
I20260620 06:12:02.983206 22860 raft_consensus.cc:3037] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:02.983165 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7947d66c6a147ad98386f6bb90a2652" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eb6c2999b44a76a4a219492b2425c2"
I20260620 06:12:02.983291 23000 raft_consensus.cc:3037] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:02.984226 22860 raft_consensus.cc:2454] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 1.
I20260620 06:12:02.984493 23071 leader_election.cc:302] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607a2035cdf140a9bbee73ce58d85868, c06c396ac802414190479a82924c56e9; no voters:
I20260620 06:12:02.984689 23222 raft_consensus.cc:2781] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:02.984710 23000 raft_consensus.cc:2454] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 1.
I20260620 06:12:02.984902 23222 raft_consensus.cc:686] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 1 LEADER]: Becoming Leader. State: Replica: c06c396ac802414190479a82924c56e9, State: Running, Role: LEADER
I20260620 06:12:02.985208 23222 consensus_queue.cc:237] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:02.987921 22701 catalog_manager.cc:5495] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c06c396ac802414190479a82924c56e9 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "c06c396ac802414190479a82924c56e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } health_report { overall_health: HEALTHY } } }
I20260620 06:12:02.999759 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:02.996548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.14.209.131:45445)
0620 06:12:02.996659 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:02.996663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:02.996674 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:12:02.996806 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:02.996808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:02.996851 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:02.996929 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:02.997160 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.997631 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.998443 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:02.998657 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:02.998732 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:02.998959 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:02.999398 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:02.999481 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:02.999576 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 06:12:02.999619 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:12:03.004936 22860 raft_consensus.cc:1261] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Refusing update from remote peer c06c396ac802414190479a82924c56e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.005079 23000 raft_consensus.cc:1261] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Refusing update from remote peer c06c396ac802414190479a82924c56e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.005656 23222 consensus_queue.cc:1035] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.005795 23224 consensus_queue.cc:1035] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.013058 23227 mvcc.cc:205] Tried to move back new op lower bound from 7298809540621430784 to 7298809540549861376. Current Snapshot: MvccSnapshot[applied={T|T < 7298809540621430784}]
I20260620 06:12:03.022076 22703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.022203 22703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.024394 22703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Sending DeleteTablet for 3 replicas of tablet d7947d66c6a147ad98386f6bb90a2652
I20260620 06:12:03.025118 22980 tablet_service.cc:1514] Processing DeleteTablet for tablet d7947d66c6a147ad98386f6bb90a2652 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:34724
I20260620 06:12:03.025440 23120 tablet_service.cc:1514] Processing DeleteTablet for tablet d7947d66c6a147ad98386f6bb90a2652 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260620 06:12:03.025581 23237 tablet_replica.cc:330] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: stopping tablet replica
I20260620 06:12:03.025704 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet d7947d66c6a147ad98386f6bb90a2652 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:35292
I20260620 06:12:03.025849 23237 raft_consensus.cc:2229] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.026114 23237 raft_consensus.cc:2258] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.026180 23239 tablet_replica.cc:330] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: stopping tablet replica
I20260620 06:12:03.026141 22703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:12:03.026489 23239 raft_consensus.cc:2229] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 06:12:03.026535 22703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:03.026749 23239 raft_consensus.cc:2258] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.027854 23237 ts_tablet_manager.cc:1905] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.027995 23239 ts_tablet_manager.cc:1905] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.030491 23237 ts_tablet_manager.cc:1918] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:03.030639 23237 log.cc:1196] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/d7947d66c6a147ad98386f6bb90a2652
I20260620 06:12:03.030652 23239 ts_tablet_manager.cc:1918] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:03.030822 23239 log.cc:1196] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/d7947d66c6a147ad98386f6bb90a2652
I20260620 06:12:03.030997 23237 ts_tablet_manager.cc:1939] T d7947d66c6a147ad98386f6bb90a2652 P 56eb6c2999b44a76a4a219492b2425c2: Deleting consensus metadata
I20260620 06:12:03.031282 23239 ts_tablet_manager.cc:1939] T d7947d66c6a147ad98386f6bb90a2652 P 607a2035cdf140a9bbee73ce58d85868: Deleting consensus metadata
I20260620 06:12:03.032262 22840 tablet_service.cc:1467] Processing CreateTablet for tablet a3a9fe817fde47d8868a20bd56e3042c (DEFAULT_TABLE table=test-table [id=6e4983937af24cd3954040644217a66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.032524 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a9fe817fde47d8868a20bd56e3042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.034052 23214 tablet_bootstrap.cc:492] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Bootstrap starting.
I20260620 06:12:03.034253 22980 tablet_service.cc:1467] Processing CreateTablet for tablet a3a9fe817fde47d8868a20bd56e3042c (DEFAULT_TABLE table=test-table [id=6e4983937af24cd3954040644217a66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.034483 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a9fe817fde47d8868a20bd56e3042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.034967 23214 tablet_bootstrap.cc:654] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.035459 22687 catalog_manager.cc:4841] TS 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509): tablet d7947d66c6a147ad98386f6bb90a2652 (table test-table [id=4c85aebd3cf3492da4bafd3e549299cd]) successfully deleted
I20260620 06:12:03.035676 22687 catalog_manager.cc:4841] TS 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159): tablet d7947d66c6a147ad98386f6bb90a2652 (table test-table [id=4c85aebd3cf3492da4bafd3e549299cd]) successfully deleted
I20260620 06:12:03.036085 23238 tablet_replica.cc:330] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: stopping tablet replica
I20260620 06:12:03.036269 23214 tablet_bootstrap.cc:492] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: No bootstrap required, opened a new log
I20260620 06:12:03.036351 23214 ts_tablet_manager.cc:1397] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:03.036393 23238 raft_consensus.cc:2229] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:12:03.036789 23238 raft_consensus.cc:2258] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.036811 23215 tablet_bootstrap.cc:492] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Bootstrap starting.
I20260620 06:12:03.036777 23214 raft_consensus.cc:348] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.036902 23214 raft_consensus.cc:374] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.036944 23214 raft_consensus.cc:729] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607a2035cdf140a9bbee73ce58d85868, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.037065 23214 consensus_queue.cc:260] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.037318 23214 ts_tablet_manager.cc:1428] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:03.037696 23215 tablet_bootstrap.cc:654] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.038286 23238 ts_tablet_manager.cc:1905] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.038589 23120 tablet_service.cc:1467] Processing CreateTablet for tablet a3a9fe817fde47d8868a20bd56e3042c (DEFAULT_TABLE table=test-table [id=6e4983937af24cd3954040644217a66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.038833 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a9fe817fde47d8868a20bd56e3042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.040741 23238 ts_tablet_manager.cc:1918] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:03.040887 23238 log.cc:1196] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/d7947d66c6a147ad98386f6bb90a2652
I20260620 06:12:03.041131 23216 tablet_bootstrap.cc:492] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Bootstrap starting.
I20260620 06:12:03.041177 23238 ts_tablet_manager.cc:1939] T d7947d66c6a147ad98386f6bb90a2652 P c06c396ac802414190479a82924c56e9: Deleting consensus metadata
I20260620 06:12:03.041850 23216 tablet_bootstrap.cc:654] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.042172 22688 catalog_manager.cc:4841] TS c06c396ac802414190479a82924c56e9 (127.14.209.131:45445): tablet d7947d66c6a147ad98386f6bb90a2652 (table test-table [id=4c85aebd3cf3492da4bafd3e549299cd]) successfully deleted
I20260620 06:12:03.042786 23215 tablet_bootstrap.cc:492] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: No bootstrap required, opened a new log
I20260620 06:12:03.042886 23215 ts_tablet_manager.cc:1397] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260620 06:12:03.043305 23215 raft_consensus.cc:348] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.043414 23215 raft_consensus.cc:374] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.043465 23215 raft_consensus.cc:729] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.043586 23215 consensus_queue.cc:260] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.043812 23215 ts_tablet_manager.cc:1428] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:03.044353 23216 tablet_bootstrap.cc:492] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: No bootstrap required, opened a new log
I20260620 06:12:03.044442 23216 ts_tablet_manager.cc:1397] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:03.044893 23216 raft_consensus.cc:348] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.044989 23216 raft_consensus.cc:374] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.045025 23216 raft_consensus.cc:729] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06c396ac802414190479a82924c56e9, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.045136 23216 consensus_queue.cc:260] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.045379 23216 ts_tablet_manager.cc:1428] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:12:03.106607 23221 raft_consensus.cc:482] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:03.106742 23221 raft_consensus.cc:504] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.108026 23221 leader_election.cc:288] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
W20260620 06:12:03.110991 23187 tablet.cc:2369] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:12:03.112763 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.108234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.14.209.130:42117)
0620 06:12:03.108427 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.108445 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.108531 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.108970 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.108972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.108987 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:03.109112 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.109118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.110143 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.110146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.111426 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.111438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.112277 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.112281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.112400 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.112465 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.112547 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.112626 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":29}
I20260620 06:12:03.112802 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.108577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:42117 (local address 127.14.209.129:37509)
0620 06:12:03.108755 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.108761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.108779 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:12:03.108831 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.108834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.108890 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.108963 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:03.109226 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.110028 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.111578 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.112163 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.112283 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.112333 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.112577 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.112678 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 06:12:03.112895 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.108359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:33297 (local address 127.14.209.131:45445)
0620 06:12:03.108473 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.108478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.108829 (+ 351us) server_negotiation.cc:407] Connection header received
0620 06:12:03.108932 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.108935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.108981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.109057 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:03.109344 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.109828 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.111585 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.112188 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.112287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.112333 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.112713 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.112781 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:12:03.113291 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.108234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45445 (local address 127.14.209.130:33297)
0620 06:12:03.108727 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.108740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.108841 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.109094 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.109097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.109110 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:03.109229 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.109235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.109940 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.109945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.111447 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.111453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.112408 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.112412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.112569 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.112595 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.113026 (+ 431us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.113103 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":276,"thread_start_us":81,"threads_started":1}
I20260620 06:12:03.113720 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a9fe817fde47d8868a20bd56e3042c" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868" is_pre_election: true
I20260620 06:12:03.113865 22860 raft_consensus.cc:2454] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 0.
I20260620 06:12:03.113847 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a9fe817fde47d8868a20bd56e3042c" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06c396ac802414190479a82924c56e9" is_pre_election: true
I20260620 06:12:03.114080 23140 raft_consensus.cc:2454] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 0.
I20260620 06:12:03.114275 22931 leader_election.cc:302] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eb6c2999b44a76a4a219492b2425c2, 607a2035cdf140a9bbee73ce58d85868; no voters:
I20260620 06:12:03.114523 23221 raft_consensus.cc:2781] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:03.114588 23221 raft_consensus.cc:482] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:03.114642 23221 raft_consensus.cc:3037] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.115903 23221 raft_consensus.cc:504] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.116498 23221 leader_election.cc:288] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 election: Requested vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
I20260620 06:12:03.116596 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a9fe817fde47d8868a20bd56e3042c" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06c396ac802414190479a82924c56e9"
I20260620 06:12:03.116602 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a9fe817fde47d8868a20bd56e3042c" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868"
I20260620 06:12:03.116712 23140 raft_consensus.cc:3037] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.116711 22860 raft_consensus.cc:3037] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.117765 22860 raft_consensus.cc:2454] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 1.
I20260620 06:12:03.117765 23140 raft_consensus.cc:2454] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 1.
I20260620 06:12:03.118034 22932 leader_election.cc:302] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eb6c2999b44a76a4a219492b2425c2, c06c396ac802414190479a82924c56e9; no voters:
I20260620 06:12:03.118198 23221 raft_consensus.cc:2781] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:03.118366 23221 raft_consensus.cc:686] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 1 LEADER]: Becoming Leader. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Running, Role: LEADER
I20260620 06:12:03.118602 23221 consensus_queue.cc:237] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.121279 22703 catalog_manager.cc:5495] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130). New cstate: current_term: 1 leader_uuid: "56eb6c2999b44a76a4a219492b2425c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } health_report { overall_health: UNKNOWN } } }
W20260620 06:12:03.152901 23047 tablet.cc:2369] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935922, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.190@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:03.181890 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.172673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.14.209.190:45585)
0620 06:12:03.172803 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.172808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.172991 (+ 183us) server_negotiation.cc:407] Connection header received
0620 06:12:03.173102 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.173104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.173141 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.173209 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:03.173974 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.174495 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.175867 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.176088 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.176169 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.178705 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:03.178708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:03.178710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:03.178730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:03.180194 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:03.180682 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:03.180684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:03.180686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:03.180754 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:03.181018 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:03.181020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:03.181022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:03.181210 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:03.181287 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.181617 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.181761 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":54}
I20260620 06:12:03.181913 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.172607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45585 (local address 127.0.0.1:41238)
0620 06:12:03.172892 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.172906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.173022 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.173251 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.173253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.173702 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:03.173807 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.173814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.174608 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.174611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.175734 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.175745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.176196 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.176199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.176290 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.176920 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:03.176935 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:03.178559 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:03.180302 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:03.180310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:03.180316 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:03.180587 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:03.180841 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:03.180844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:03.180845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:03.180930 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:03.181317 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:03.181323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:03.181439 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.181562 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.181709 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
W20260620 06:12:03.184134 22907 tablet.cc:2369] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:12:03.185951 22703 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:41238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:12:03.186676 22703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:03.191250 22840 tablet_service.cc:1467] Processing CreateTablet for tablet 568de2d78b7a4b6ba6a97b1fcd524f19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dbdf9c939494e609d8cf8cc78d6c552]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:12:03.191296 23120 tablet_service.cc:1467] Processing CreateTablet for tablet 568de2d78b7a4b6ba6a97b1fcd524f19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dbdf9c939494e609d8cf8cc78d6c552]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:12:03.191325 22980 tablet_service.cc:1467] Processing CreateTablet for tablet 568de2d78b7a4b6ba6a97b1fcd524f19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dbdf9c939494e609d8cf8cc78d6c552]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:12:03.191614 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568de2d78b7a4b6ba6a97b1fcd524f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.191613 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568de2d78b7a4b6ba6a97b1fcd524f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.191706 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568de2d78b7a4b6ba6a97b1fcd524f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.194214 23215 tablet_bootstrap.cc:492] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: Bootstrap starting.
I20260620 06:12:03.194270 23216 tablet_bootstrap.cc:492] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: Bootstrap starting.
I20260620 06:12:03.194824 23214 tablet_bootstrap.cc:492] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: Bootstrap starting.
I20260620 06:12:03.195199 23215 tablet_bootstrap.cc:654] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.195212 23216 tablet_bootstrap.cc:654] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.195871 23214 tablet_bootstrap.cc:654] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.196210 23216 tablet_bootstrap.cc:492] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: No bootstrap required, opened a new log
I20260620 06:12:03.196278 23216 ts_tablet_manager.cc:1397] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:12:03.196691 23216 raft_consensus.cc:348] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.196816 23216 raft_consensus.cc:374] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.196853 23216 raft_consensus.cc:729] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06c396ac802414190479a82924c56e9, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.196967 23216 consensus_queue.cc:260] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.197021 23215 tablet_bootstrap.cc:492] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: No bootstrap required, opened a new log
I20260620 06:12:03.197091 23215 ts_tablet_manager.cc:1397] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 06:12:03.197227 23216 ts_tablet_manager.cc:1428] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:12:03.197247 23214 tablet_bootstrap.cc:492] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: No bootstrap required, opened a new log
I20260620 06:12:03.197312 23214 ts_tablet_manager.cc:1397] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:03.197453 23232 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.197585 23232 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.197522 23215 raft_consensus.cc:348] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.197626 23215 raft_consensus.cc:374] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.197659 23215 raft_consensus.cc:729] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.197723 23214 raft_consensus.cc:348] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.197818 23214 raft_consensus.cc:374] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.197778 23215 consensus_queue.cc:260] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.197849 23214 raft_consensus.cc:729] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607a2035cdf140a9bbee73ce58d85868, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.197956 23214 consensus_queue.cc:260] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.198040 23215 ts_tablet_manager.cc:1428] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 06:12:03.198200 23214 ts_tablet_manager.cc:1428] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:03.198239 23220 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.198364 23220 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.198643 23221 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.198761 23221 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.229637 23220 raft_consensus.cc:482] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:03.229755 23220 raft_consensus.cc:504] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.230826 23220 leader_election.cc:288] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159), c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
I20260620 06:12:03.236310 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.231124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:49307 (local address 127.14.209.130:41159)
0620 06:12:03.231258 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.231265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.231528 (+ 263us) server_negotiation.cc:407] Connection header received
0620 06:12:03.231671 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.231674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.231724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.231807 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:03.232301 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.233108 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.234974 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.235578 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.235705 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.235753 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.236119 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.236187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:12:03.236327 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.231199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:33771 (local address 127.14.209.131:45445)
0620 06:12:03.231325 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.231331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.231588 (+ 257us) server_negotiation.cc:407] Connection header received
0620 06:12:03.231692 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.231694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.231736 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.231807 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:03.232301 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.233084 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.234936 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.235578 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.235705 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.235754 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.236119 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.236188 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:12:03.236357 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.231024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:41159 (local address 127.14.209.129:49307)
0620 06:12:03.231426 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.231441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.231556 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.231830 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.231833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.231870 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:03.232151 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.232160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.233238 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.233242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.234785 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.234793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.235734 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.235738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.235963 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.236009 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.236089 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.236164 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":263,"spinlock_wait_cycles":31104,"thread_start_us":140,"threads_started":1}
I20260620 06:12:03.236361 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.231090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45445 (local address 127.14.209.129:33771)
0620 06:12:03.231496 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.231509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.231588 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.231830 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.231833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.231871 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:03.232171 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.232177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.233238 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.233242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.234781 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.234793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.235734 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.235738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.235959 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.236006 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.236089 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.236164 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":209,"spinlock_wait_cycles":6272,"thread_start_us":105,"threads_started":1}
I20260620 06:12:03.236861 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568de2d78b7a4b6ba6a97b1fcd524f19" candidate_uuid: "607a2035cdf140a9bbee73ce58d85868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eb6c2999b44a76a4a219492b2425c2" is_pre_election: true
I20260620 06:12:03.237036 23000 raft_consensus.cc:2454] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607a2035cdf140a9bbee73ce58d85868 in term 0.
I20260620 06:12:03.237138 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568de2d78b7a4b6ba6a97b1fcd524f19" candidate_uuid: "607a2035cdf140a9bbee73ce58d85868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06c396ac802414190479a82924c56e9" is_pre_election: true
I20260620 06:12:03.237272 23140 raft_consensus.cc:2454] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607a2035cdf140a9bbee73ce58d85868 in term 0.
I20260620 06:12:03.237370 22791 leader_election.cc:302] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eb6c2999b44a76a4a219492b2425c2, 607a2035cdf140a9bbee73ce58d85868; no voters:
I20260620 06:12:03.237628 23220 raft_consensus.cc:2781] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:03.237699 23220 raft_consensus.cc:482] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:03.237778 23220 raft_consensus.cc:3037] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.238765 23220 raft_consensus.cc:504] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.239183 23220 leader_election.cc:288] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [CANDIDATE]: Term 1 election: Requested vote from peers 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159), c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
I20260620 06:12:03.239511 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568de2d78b7a4b6ba6a97b1fcd524f19" candidate_uuid: "607a2035cdf140a9bbee73ce58d85868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06c396ac802414190479a82924c56e9"
I20260620 06:12:03.239534 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568de2d78b7a4b6ba6a97b1fcd524f19" candidate_uuid: "607a2035cdf140a9bbee73ce58d85868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eb6c2999b44a76a4a219492b2425c2"
I20260620 06:12:03.239640 23000 raft_consensus.cc:3037] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.239640 23140 raft_consensus.cc:3037] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.240679 23000 raft_consensus.cc:2454] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607a2035cdf140a9bbee73ce58d85868 in term 1.
I20260620 06:12:03.240679 23140 raft_consensus.cc:2454] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607a2035cdf140a9bbee73ce58d85868 in term 1.
I20260620 06:12:03.241006 22791 leader_election.cc:302] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eb6c2999b44a76a4a219492b2425c2, 607a2035cdf140a9bbee73ce58d85868; no voters:
I20260620 06:12:03.241183 23220 raft_consensus.cc:2781] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:03.241370 23220 raft_consensus.cc:686] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [term 1 LEADER]: Becoming Leader. State: Replica: 607a2035cdf140a9bbee73ce58d85868, State: Running, Role: LEADER
I20260620 06:12:03.241647 23220 consensus_queue.cc:237] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.243618 23257 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: TxnStatusTablet state changed. Reason: New leader 607a2035cdf140a9bbee73ce58d85868. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.243738 23257 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:12:03.243991 23259 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:12:03.244146 23259 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:12:03.244112 22703 catalog_manager.cc:5495] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129). New cstate: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20260620 06:12:03.264770 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.260802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.0.0.1:35302)
0620 06:12:03.260926 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.260943 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.261037 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.261376 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.261378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.261386 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:12:03.261473 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.261478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.262474 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.262476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.263149 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.263158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.263720 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.263723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.263798 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.263837 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.264531 (+ 694us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.264544 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.264605 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.264669 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 06:12:03.264791 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.260876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.14.209.129:37509)
0620 06:12:03.260995 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.260999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.261014 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:12:03.261166 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.261169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.261232 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.261335 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.261585 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.262363 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.263256 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.263598 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.263699 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.263969 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.264422 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.264540 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.264619 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.264668 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:12:03.267601 23000 raft_consensus.cc:1261] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Refusing update from remote peer 607a2035cdf140a9bbee73ce58d85868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.267822 23140 raft_consensus.cc:1261] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Refusing update from remote peer 607a2035cdf140a9bbee73ce58d85868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.268023 23220 consensus_queue.cc:1035] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.268251 23257 consensus_queue.cc:1035] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.269675 23221 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: TxnStatusTablet state changed. Reason: New leader 607a2035cdf140a9bbee73ce58d85868. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.269783 23221 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.269790 23232 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: TxnStatusTablet state changed. Reason: New leader 607a2035cdf140a9bbee73ce58d85868. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.269896 23232 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.271710 23257 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.271809 23257 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:12:03.272965 23232 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.273082 23232 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P c06c396ac802414190479a82924c56e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.275985 23220 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.276096 23220 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 607a2035cdf140a9bbee73ce58d85868: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:12:03.276821 23221 tablet_replica.cc:439] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "607a2035cdf140a9bbee73ce58d85868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } } }
I20260620 06:12:03.276919 23221 tablet_replica.cc:442] T 568de2d78b7a4b6ba6a97b1fcd524f19 P 56eb6c2999b44a76a4a219492b2425c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:12:03.281054 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.276904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.14.209.130:41159)
0620 06:12:03.277002 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.277006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.277017 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:12:03.277091 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.277096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.277150 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.277269 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.277504 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.278296 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.279447 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.279791 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.279899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.280169 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.280662 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.280772 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.280877 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.280929 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:12:03.291651 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.287454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:55545 (local address 127.14.209.129:37509)
0620 06:12:03.287573 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.287578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.287821 (+ 243us) server_negotiation.cc:407] Connection header received
0620 06:12:03.287930 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.287932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.287968 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.288026 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.288267 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.288722 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.290644 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.290850 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.290936 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.291278 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.291342 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.291400 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.291509 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.291550 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:12:03.291738 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.287345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.14.209.130:55545)
0620 06:12:03.287733 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.287747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.287843 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.288056 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.288059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.288066 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:12:03.288173 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.288178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.288815 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.288819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.290487 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.290498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.290982 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.290986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.291100 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.291148 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.291426 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.291432 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.291503 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.291587 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":96,"threads_started":1}
I20260620 06:12:03.298722 22860 raft_consensus.cc:1261] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Refusing update from remote peer 56eb6c2999b44a76a4a219492b2425c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.298707 23140 raft_consensus.cc:1261] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Refusing update from remote peer 56eb6c2999b44a76a4a219492b2425c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.299196 23244 consensus_queue.cc:1035] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.299408 23221 consensus_queue.cc:1035] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.303194 23229 mvcc.cc:205] Tried to move back new op lower bound from 7298809541826830336 to 7298809541096312832. Current Snapshot: MvccSnapshot[applied={T|T < 7298809541826830336}]
I20260620 06:12:03.305151 23228 mvcc.cc:205] Tried to move back new op lower bound from 7298809541826830336 to 7298809541096312832. Current Snapshot: MvccSnapshot[applied={T|T < 7298809541826830336}]
I20260620 06:12:03.320403 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.316819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:57101 (local address 127.14.209.130:41159)
0620 06:12:03.316924 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.316934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.317189 (+ 255us) server_negotiation.cc:407] Connection header received
0620 06:12:03.317307 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.317309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.317347 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.317414 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.317703 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.318268 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.319459 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.319667 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.319744 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.320050 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.320110 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.320158 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.320271 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.320310 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 06:12:03.320526 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.316737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:41159 (local address 127.14.209.129:57101)
0620 06:12:03.317107 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.317122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.317225 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.317446 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.317449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.317456 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:12:03.317578 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.317614 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.318376 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.318379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.319353 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.319364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.319783 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.319786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.319879 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.319927 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.320189 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.320194 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.320265 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.320335 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260620 06:12:03.345772 22702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.345930 22702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.348093 22702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Sending DeleteTablet for 3 replicas of tablet a3a9fe817fde47d8868a20bd56e3042c
I20260620 06:12:03.348949 22980 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a9fe817fde47d8868a20bd56e3042c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:34724
I20260620 06:12:03.349134 23237 tablet_replica.cc:330] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: stopping tablet replica
I20260620 06:12:03.349258 23237 raft_consensus.cc:2229] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:12:03.349277 23120 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a9fe817fde47d8868a20bd56e3042c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260620 06:12:03.349437 23238 tablet_replica.cc:330] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: stopping tablet replica
I20260620 06:12:03.349444 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a9fe817fde47d8868a20bd56e3042c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:35292
I20260620 06:12:03.349519 23237 raft_consensus.cc:2258] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.349548 23238 raft_consensus.cc:2229] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.349586 23239 tablet_replica.cc:330] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: stopping tablet replica
I20260620 06:12:03.349642 23238 raft_consensus.cc:2258] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.349677 23239 raft_consensus.cc:2229] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.349776 23239 raft_consensus.cc:2258] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.350564 23239 ts_tablet_manager.cc:1905] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.350584 23238 ts_tablet_manager.cc:1905] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.353287 23239 ts_tablet_manager.cc:1918] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:12:03.353309 23238 ts_tablet_manager.cc:1918] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:12:03.353353 23239 log.cc:1196] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/a3a9fe817fde47d8868a20bd56e3042c
I20260620 06:12:03.353374 23238 log.cc:1196] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/a3a9fe817fde47d8868a20bd56e3042c
I20260620 06:12:03.353611 23239 ts_tablet_manager.cc:1939] T a3a9fe817fde47d8868a20bd56e3042c P 607a2035cdf140a9bbee73ce58d85868: Deleting consensus metadata
I20260620 06:12:03.353618 23238 ts_tablet_manager.cc:1939] T a3a9fe817fde47d8868a20bd56e3042c P c06c396ac802414190479a82924c56e9: Deleting consensus metadata
I20260620 06:12:03.354281 23237 ts_tablet_manager.cc:1905] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.354286 22688 catalog_manager.cc:4841] TS c06c396ac802414190479a82924c56e9 (127.14.209.131:45445): tablet a3a9fe817fde47d8868a20bd56e3042c (table test-table [id=6e4983937af24cd3954040644217a66e]) successfully deleted
I20260620 06:12:03.354286 22687 catalog_manager.cc:4841] TS 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509): tablet a3a9fe817fde47d8868a20bd56e3042c (table test-table [id=6e4983937af24cd3954040644217a66e]) successfully deleted
I20260620 06:12:03.356920 23237 ts_tablet_manager.cc:1918] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:12:03.356980 23237 log.cc:1196] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/a3a9fe817fde47d8868a20bd56e3042c
I20260620 06:12:03.357208 23237 ts_tablet_manager.cc:1939] T a3a9fe817fde47d8868a20bd56e3042c P 56eb6c2999b44a76a4a219492b2425c2: Deleting consensus metadata
I20260620 06:12:03.358063 22687 catalog_manager.cc:4841] TS 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159): tablet a3a9fe817fde47d8868a20bd56e3042c (table test-table [id=6e4983937af24cd3954040644217a66e]) successfully deleted
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:12:03.389568 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:03.381792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.14.209.190:45585)
0620 06:12:03.381907 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.381911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.382258 (+ 347us) server_negotiation.cc:407] Connection header received
0620 06:12:03.382378 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.382382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.382432 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.382514 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:03.383421 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.384231 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.385417 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.385761 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.385863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.389442 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41244: BlockingRecv error: recv got EOF from 127.0.0.1:41244 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM
I20260620 06:12:03.400951 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.391006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.14.209.190:45585)
0620 06:12:03.391139 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.391143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.391561 (+ 418us) server_negotiation.cc:407] Connection header received
0620 06:12:03.391665 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.391668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.391710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.391788 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:03.392481 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.393235 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.394345 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.394675 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.394774 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.397526 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:03.397530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:03.397533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:03.397557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:03.399253 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:03.399717 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:03.399720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:03.399722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:03.399808 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:03.400110 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:03.400113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:03.400115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:03.400323 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:03.400464 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.400699 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.400803 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:12:03.403585 22702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:12:03.404032 22702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:03.408855 23120 tablet_service.cc:1467] Processing CreateTablet for tablet b9209833db42481fa0ea3b77960d6d5a (DEFAULT_TABLE table=test-table [id=bd9704c0ea4246fbadcaecf5a88a8661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.408870 22840 tablet_service.cc:1467] Processing CreateTablet for tablet b9209833db42481fa0ea3b77960d6d5a (DEFAULT_TABLE table=test-table [id=bd9704c0ea4246fbadcaecf5a88a8661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.409123 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9209833db42481fa0ea3b77960d6d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.409134 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9209833db42481fa0ea3b77960d6d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.411306 22980 tablet_service.cc:1467] Processing CreateTablet for tablet b9209833db42481fa0ea3b77960d6d5a (DEFAULT_TABLE table=test-table [id=bd9704c0ea4246fbadcaecf5a88a8661]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.411453 23216 tablet_bootstrap.cc:492] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Bootstrap starting.
I20260620 06:12:03.411545 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9209833db42481fa0ea3b77960d6d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.412273 23216 tablet_bootstrap.cc:654] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.413960 23215 tablet_bootstrap.cc:492] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Bootstrap starting.
I20260620 06:12:03.414284 23216 tablet_bootstrap.cc:492] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: No bootstrap required, opened a new log
I20260620 06:12:03.414352 23216 ts_tablet_manager.cc:1397] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 06:12:03.414739 23216 raft_consensus.cc:348] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.414837 23216 raft_consensus.cc:374] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.414851 23215 tablet_bootstrap.cc:654] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.414865 23216 raft_consensus.cc:729] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06c396ac802414190479a82924c56e9, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.414994 23216 consensus_queue.cc:260] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.415247 23216 ts_tablet_manager.cc:1428] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:12:03.416112 23214 tablet_bootstrap.cc:492] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Bootstrap starting.
I20260620 06:12:03.417025 23214 tablet_bootstrap.cc:654] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.418087 23215 tablet_bootstrap.cc:492] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: No bootstrap required, opened a new log
I20260620 06:12:03.418177 23215 ts_tablet_manager.cc:1397] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:12:03.418509 23214 tablet_bootstrap.cc:492] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: No bootstrap required, opened a new log
I20260620 06:12:03.418574 23214 ts_tablet_manager.cc:1397] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:03.418550 23215 raft_consensus.cc:348] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.418659 23215 raft_consensus.cc:374] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.418691 23215 raft_consensus.cc:729] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.418802 23215 consensus_queue.cc:260] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.418985 23214 raft_consensus.cc:348] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.419162 23214 raft_consensus.cc:374] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.419195 23214 raft_consensus.cc:729] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607a2035cdf140a9bbee73ce58d85868, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.419329 23214 consensus_queue.cc:260] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.419021 23215 ts_tablet_manager.cc:1428] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:03.420056 23214 ts_tablet_manager.cc:1428] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:12:03.434433 23244 raft_consensus.cc:482] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:03.434574 23244 raft_consensus.cc:504] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.434959 23244 leader_election.cc:288] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
I20260620 06:12:03.435359 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9209833db42481fa0ea3b77960d6d5a" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868" is_pre_election: true
I20260620 06:12:03.435369 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9209833db42481fa0ea3b77960d6d5a" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06c396ac802414190479a82924c56e9" is_pre_election: true
I20260620 06:12:03.435491 22860 raft_consensus.cc:2454] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 0.
I20260620 06:12:03.435503 23140 raft_consensus.cc:2454] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 0.
I20260620 06:12:03.435791 22931 leader_election.cc:302] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eb6c2999b44a76a4a219492b2425c2, 607a2035cdf140a9bbee73ce58d85868; no voters:
I20260620 06:12:03.435962 23244 raft_consensus.cc:2781] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:03.436017 23244 raft_consensus.cc:482] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:03.436069 23244 raft_consensus.cc:3037] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.437247 23244 raft_consensus.cc:504] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.437668 23244 leader_election.cc:288] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 election: Requested vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), c06c396ac802414190479a82924c56e9 (127.14.209.131:45445)
I20260620 06:12:03.437947 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9209833db42481fa0ea3b77960d6d5a" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868"
I20260620 06:12:03.437986 23140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9209833db42481fa0ea3b77960d6d5a" candidate_uuid: "56eb6c2999b44a76a4a219492b2425c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06c396ac802414190479a82924c56e9"
I20260620 06:12:03.438061 22860 raft_consensus.cc:3037] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.438074 23140 raft_consensus.cc:3037] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.439121 22860 raft_consensus.cc:2454] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 1.
I20260620 06:12:03.439435 22931 leader_election.cc:302] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56eb6c2999b44a76a4a219492b2425c2, 607a2035cdf140a9bbee73ce58d85868; no voters:
I20260620 06:12:03.439483 23140 raft_consensus.cc:2454] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56eb6c2999b44a76a4a219492b2425c2 in term 1.
I20260620 06:12:03.439899 23244 raft_consensus.cc:2781] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:03.439992 23244 raft_consensus.cc:686] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 1 LEADER]: Becoming Leader. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Running, Role: LEADER
I20260620 06:12:03.440111 23244 consensus_queue.cc:237] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.441753 22702 catalog_manager.cc:5495] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130). New cstate: current_term: 1 leader_uuid: "56eb6c2999b44a76a4a219492b2425c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } health_report { overall_health: UNKNOWN } } }
I20260620 06:12:03.454259 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.451131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.14.209.130:41159)
0620 06:12:03.451490 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.451495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.451507 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:12:03.451546 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.451549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.451595 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.451671 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.451916 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.452416 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.453250 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.453478 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.453586 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.453825 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.453895 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.453972 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.454082 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.454128 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260620 06:12:03.455976 23140 raft_consensus.cc:1261] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Refusing update from remote peer 56eb6c2999b44a76a4a219492b2425c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.455976 22860 raft_consensus.cc:1261] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Refusing update from remote peer 56eb6c2999b44a76a4a219492b2425c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.456401 23244 consensus_queue.cc:1035] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.456524 23270 consensus_queue.cc:1035] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.462909 22703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.463035 22703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.465525 22703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Sending DeleteTablet for 3 replicas of tablet b9209833db42481fa0ea3b77960d6d5a
I20260620 06:12:03.466154 22980 tablet_service.cc:1514] Processing DeleteTablet for tablet b9209833db42481fa0ea3b77960d6d5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:34724
I20260620 06:12:03.466157 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet b9209833db42481fa0ea3b77960d6d5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:35292
I20260620 06:12:03.466236 23120 tablet_service.cc:1514] Processing DeleteTablet for tablet b9209833db42481fa0ea3b77960d6d5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260620 06:12:03.466315 23239 tablet_replica.cc:330] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: stopping tablet replica
I20260620 06:12:03.466390 23237 tablet_replica.cc:330] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: stopping tablet replica
I20260620 06:12:03.466413 23239 raft_consensus.cc:2229] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.466480 23237 raft_consensus.cc:2229] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:12:03.466519 23239 raft_consensus.cc:2258] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.466655 23237 raft_consensus.cc:2258] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.466807 23238 tablet_replica.cc:330] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: stopping tablet replica
I20260620 06:12:03.466915 23238 raft_consensus.cc:2229] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.467029 23238 raft_consensus.cc:2258] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.467213 23239 ts_tablet_manager.cc:1905] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.467670 23238 ts_tablet_manager.cc:1905] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.468626 22703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:12:03.469218 22703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:03.469666 23239 ts_tablet_manager.cc:1918] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:03.469743 23239 log.cc:1196] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/b9209833db42481fa0ea3b77960d6d5a
I20260620 06:12:03.469995 23239 ts_tablet_manager.cc:1939] T b9209833db42481fa0ea3b77960d6d5a P 607a2035cdf140a9bbee73ce58d85868: Deleting consensus metadata
I20260620 06:12:03.470008 23238 ts_tablet_manager.cc:1918] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:03.470067 23238 log.cc:1196] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/b9209833db42481fa0ea3b77960d6d5a
I20260620 06:12:03.470295 23238 ts_tablet_manager.cc:1939] T b9209833db42481fa0ea3b77960d6d5a P c06c396ac802414190479a82924c56e9: Deleting consensus metadata
I20260620 06:12:03.470664 22687 catalog_manager.cc:4841] TS 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509): tablet b9209833db42481fa0ea3b77960d6d5a (table test-table [id=bd9704c0ea4246fbadcaecf5a88a8661]) successfully deleted
I20260620 06:12:03.470862 22688 catalog_manager.cc:4841] TS c06c396ac802414190479a82924c56e9 (127.14.209.131:45445): tablet b9209833db42481fa0ea3b77960d6d5a (table test-table [id=bd9704c0ea4246fbadcaecf5a88a8661]) successfully deleted
I20260620 06:12:03.471716 23237 ts_tablet_manager.cc:1905] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.473711 23237 ts_tablet_manager.cc:1918] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:03.473774 23237 log.cc:1196] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/b9209833db42481fa0ea3b77960d6d5a
I20260620 06:12:03.474018 23237 ts_tablet_manager.cc:1939] T b9209833db42481fa0ea3b77960d6d5a P 56eb6c2999b44a76a4a219492b2425c2: Deleting consensus metadata
I20260620 06:12:03.474606 22840 tablet_service.cc:1467] Processing CreateTablet for tablet 6b21400c85234cb282c632b00a74fc34 (DEFAULT_TABLE table=test-table [id=1d5028576fa94ff985bc4eeb0bbd609f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.474625 23120 tablet_service.cc:1467] Processing CreateTablet for tablet 6b21400c85234cb282c632b00a74fc34 (DEFAULT_TABLE table=test-table [id=1d5028576fa94ff985bc4eeb0bbd609f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.474852 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b21400c85234cb282c632b00a74fc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.474892 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b21400c85234cb282c632b00a74fc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.477008 23216 tablet_bootstrap.cc:492] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Bootstrap starting.
I20260620 06:12:03.477097 23214 tablet_bootstrap.cc:492] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: Bootstrap starting.
I20260620 06:12:03.477808 23214 tablet_bootstrap.cc:654] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.477808 23216 tablet_bootstrap.cc:654] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.479007 22980 tablet_service.cc:1467] Processing CreateTablet for tablet 6b21400c85234cb282c632b00a74fc34 (DEFAULT_TABLE table=test-table [id=1d5028576fa94ff985bc4eeb0bbd609f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:03.479159 23216 tablet_bootstrap.cc:492] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: No bootstrap required, opened a new log
I20260620 06:12:03.479277 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b21400c85234cb282c632b00a74fc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:03.479266 23216 ts_tablet_manager.cc:1397] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:12:03.479705 23216 raft_consensus.cc:348] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.479818 23216 raft_consensus.cc:374] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.479851 23216 raft_consensus.cc:729] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06c396ac802414190479a82924c56e9, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.479971 23216 consensus_queue.cc:260] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.480232 23216 ts_tablet_manager.cc:1428] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:12:03.481328 23215 tablet_bootstrap.cc:492] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Bootstrap starting.
I20260620 06:12:03.482161 22687 catalog_manager.cc:4841] TS 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159): tablet b9209833db42481fa0ea3b77960d6d5a (table test-table [id=bd9704c0ea4246fbadcaecf5a88a8661]) successfully deleted
I20260620 06:12:03.482214 23215 tablet_bootstrap.cc:654] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:03.482679 23214 tablet_bootstrap.cc:492] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: No bootstrap required, opened a new log
I20260620 06:12:03.482755 23214 ts_tablet_manager.cc:1397] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 06:12:03.483222 23214 raft_consensus.cc:348] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.483340 23214 raft_consensus.cc:374] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.483371 23214 raft_consensus.cc:729] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607a2035cdf140a9bbee73ce58d85868, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.483484 23214 consensus_queue.cc:260] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.483744 23214 ts_tablet_manager.cc:1428] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:12:03.484902 23215 tablet_bootstrap.cc:492] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: No bootstrap required, opened a new log
I20260620 06:12:03.484993 23215 ts_tablet_manager.cc:1397] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 06:12:03.485404 23215 raft_consensus.cc:348] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.485505 23215 raft_consensus.cc:374] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:03.485527 23215 raft_consensus.cc:729] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eb6c2999b44a76a4a219492b2425c2, State: Initialized, Role: FOLLOWER
I20260620 06:12:03.485656 23215 consensus_queue.cc:260] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.485951 23215 ts_tablet_manager.cc:1428] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:12:03.752539 23232 raft_consensus.cc:482] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:03.752686 23232 raft_consensus.cc:504] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.753119 23232 leader_election.cc:288] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159)
I20260620 06:12:03.753644 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21400c85234cb282c632b00a74fc34" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eb6c2999b44a76a4a219492b2425c2" is_pre_election: true
I20260620 06:12:03.753649 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21400c85234cb282c632b00a74fc34" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868" is_pre_election: true
I20260620 06:12:03.753801 23000 raft_consensus.cc:2454] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 0.
I20260620 06:12:03.753801 22860 raft_consensus.cc:2454] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 0.
I20260620 06:12:03.754099 23071 leader_election.cc:302] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607a2035cdf140a9bbee73ce58d85868, c06c396ac802414190479a82924c56e9; no voters:
I20260620 06:12:03.754288 23232 raft_consensus.cc:2781] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:03.754349 23232 raft_consensus.cc:482] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:03.754390 23232 raft_consensus.cc:3037] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.755564 23232 raft_consensus.cc:504] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.755951 23232 leader_election.cc:288] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 election: Requested vote from peers 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509), 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159)
I20260620 06:12:03.756296 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21400c85234cb282c632b00a74fc34" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607a2035cdf140a9bbee73ce58d85868"
I20260620 06:12:03.756314 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b21400c85234cb282c632b00a74fc34" candidate_uuid: "c06c396ac802414190479a82924c56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eb6c2999b44a76a4a219492b2425c2"
I20260620 06:12:03.756419 23000 raft_consensus.cc:3037] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.756434 22860 raft_consensus.cc:3037] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:03.757318 22860 raft_consensus.cc:2454] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 1.
I20260620 06:12:03.757334 23000 raft_consensus.cc:2454] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06c396ac802414190479a82924c56e9 in term 1.
I20260620 06:12:03.757581 23071 leader_election.cc:302] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607a2035cdf140a9bbee73ce58d85868, c06c396ac802414190479a82924c56e9; no voters:
I20260620 06:12:03.757812 23232 raft_consensus.cc:2781] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:03.757982 23232 raft_consensus.cc:686] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 1 LEADER]: Becoming Leader. State: Replica: c06c396ac802414190479a82924c56e9, State: Running, Role: LEADER
I20260620 06:12:03.758159 23232 consensus_queue.cc:237] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } }
I20260620 06:12:03.759725 22702 catalog_manager.cc:5495] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c06c396ac802414190479a82924c56e9 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "c06c396ac802414190479a82924c56e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c06c396ac802414190479a82924c56e9" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45445 } health_report { overall_health: HEALTHY } } }
I20260620 06:12:03.774513 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.771468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.14.209.131:45445)
0620 06:12:03.771679 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.771683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.771697 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:12:03.771790 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.771793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.771839 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.771927 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.772164 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.772674 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.773501 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.773721 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.773829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.774083 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.774146 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.774225 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.774320 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.774364 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260620 06:12:03.783252 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.780408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:33877 (local address 127.14.209.129:37509)
0620 06:12:03.780527 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.780530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.780685 (+ 155us) server_negotiation.cc:407] Connection header received
0620 06:12:03.780777 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.780779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.780815 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.780875 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.781055 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.781507 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.782348 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.782589 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.782661 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.782871 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.782926 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.782976 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.783127 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.783164 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:12:03.783324 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.780351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.14.209.131:33877)
0620 06:12:03.780611 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.780624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.780704 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.780898 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.780900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.780905 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:12:03.780975 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.780978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.781587 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.781589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.782230 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.782238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.782675 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.782677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.782734 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.782768 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.783007 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.783012 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.783075 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.783179 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260620 06:12:03.788508 23000 raft_consensus.cc:1261] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Refusing update from remote peer c06c396ac802414190479a82924c56e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.788610 22860 raft_consensus.cc:1261] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Refusing update from remote peer c06c396ac802414190479a82924c56e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:03.788931 23232 consensus_queue.cc:1035] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eb6c2999b44a76a4a219492b2425c2" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.789065 23297 consensus_queue.cc:1035] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607a2035cdf140a9bbee73ce58d85868" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:03.803396 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.800168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45445 (local address 127.14.209.129:34073)
0620 06:12:03.800353 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.800366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:03.800451 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:03.800707 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:03.800714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:03.800723 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:12:03.800811 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.800815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.801430 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.801432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.801974 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:03.802006 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.802498 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.802502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:03.802579 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.802615 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.803112 (+ 497us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.803121 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:12:03.803188 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:12:03.803260 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 06:12:03.803419 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.800227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:34073 (local address 127.14.209.131:45445)
0620 06:12:03.800330 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:03.800334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:03.800409 (+ 75us) server_negotiation.cc:407] Connection header received
0620 06:12:03.800546 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:03.800550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:03.800598 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:03.800676 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:03.800892 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.801345 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.802127 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:03.802416 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:03.802489 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:03.802735 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:03.802813 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:03.802865 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:03.803211 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 06:12:03.803266 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:12:03.822140 22702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.822264 22702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:03.823961 22702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913c2dd4aaa04b629070fb4ac5e96a0b: Sending DeleteTablet for 3 replicas of tablet 6b21400c85234cb282c632b00a74fc34
I20260620 06:12:03.824416 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b21400c85234cb282c632b00a74fc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:35292
I20260620 06:12:03.824489 22980 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b21400c85234cb282c632b00a74fc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:34724
I20260620 06:12:03.824548 23120 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b21400c85234cb282c632b00a74fc34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:03 UTC) from {username='oryx', principal='oryx/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260620 06:12:03.824594 23239 tablet_replica.cc:330] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: stopping tablet replica
I20260620 06:12:03.824635 23237 tablet_replica.cc:330] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: stopping tablet replica
I20260620 06:12:03.824718 23238 tablet_replica.cc:330] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: stopping tablet replica
I20260620 06:12:03.824718 23239 raft_consensus.cc:2229] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.824743 23237 raft_consensus.cc:2229] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:03.824817 23239 raft_consensus.cc:2258] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.824834 23237 raft_consensus.cc:2258] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.824836 23238 raft_consensus.cc:2229] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:12:03.825001 23238 raft_consensus.cc:2258] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:03.825526 23237 ts_tablet_manager.cc:1905] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.825568 23239 ts_tablet_manager.cc:1905] T 6b21400c85234cb282c632b00a74fc34 P 607a2035cdf140a9bbee73ce58d85868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.825577 23238 ts_tablet_manager.cc:1905] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:03.825744 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 22775
I20260620 06:12:03.827881 23237 ts_tablet_manager.cc:1918] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:12:03.827881 23238 ts_tablet_manager.cc:1918] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:12:03.827945 23238 log.cc:1196] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/6b21400c85234cb282c632b00a74fc34
I20260620 06:12:03.827946 23237 log.cc:1196] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/6b21400c85234cb282c632b00a74fc34
I20260620 06:12:03.828200 23238 ts_tablet_manager.cc:1939] T 6b21400c85234cb282c632b00a74fc34 P c06c396ac802414190479a82924c56e9: Deleting consensus metadata
I20260620 06:12:03.828205 23237 ts_tablet_manager.cc:1939] T 6b21400c85234cb282c632b00a74fc34 P 56eb6c2999b44a76a4a219492b2425c2: Deleting consensus metadata
I20260620 06:12:03.828783 22688 catalog_manager.cc:4841] TS c06c396ac802414190479a82924c56e9 (127.14.209.131:45445): tablet 6b21400c85234cb282c632b00a74fc34 (table test-table [id=1d5028576fa94ff985bc4eeb0bbd609f]) successfully deleted
I20260620 06:12:03.829372 22687 catalog_manager.cc:4841] TS 56eb6c2999b44a76a4a219492b2425c2 (127.14.209.130:41159): tablet 6b21400c85234cb282c632b00a74fc34 (table test-table [id=1d5028576fa94ff985bc4eeb0bbd609f]) successfully deleted
W20260620 06:12:03.834501 22687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:37509 (error 108)
I20260620 06:12:03.834642 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 22915
I20260620 06:12:03.836076 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:03.835611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:37509 (local address 127.0.0.1:35310)
0620 06:12:03.835878 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:03.835956 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":56,"threads_started":1}
W20260620 06:12:03.836278 22687 catalog_manager.cc:4584] TS 607a2035cdf140a9bbee73ce58d85868 (127.14.209.129:37509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b21400c85234cb282c632b00a74fc34: Network error: Client connection negotiation failed: client connection to 127.14.209.129:37509: connect: Connection refused (error 111)
I20260620 06:12:03.841667 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23055
I20260620 06:12:03.848181 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 22671
2026-06-20T06:12:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3029 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[23311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[23311](info): set up 2 sockets
Jun 20 06:12:03 dist-test-slave-zt01 krb5kdc[23311](info): commencing operation
krb5kdc: starting...
W20260620 06:12:04.892211 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:12:04 dist-test-slave-zt01 krb5kdc[23311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:12:05.010149 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:36699
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:36699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:05.114038 23324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:05.114321 23324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:05.114393 23324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:05.118340 23324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:05.118464 23324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:05.118508 23324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:05.118567 23324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:05.118605 23324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:05.123641 23324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:36699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:05.124694 23324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:05.125663 23324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:05.134043 23329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:05.134199 23324 server_base.cc:1034] running on GCE node
W20260620 06:12:05.134053 23330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:05.134335 23332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:05.134992 23324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:05.136022 23324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:05.137250 23324 hybrid_clock.cc:648] HybridClock initialized: now 1781935925137231 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T06:12:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1298 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:12:05 dist-test-slave-zt01 krb5kdc[23337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:12:05 dist-test-slave-zt01 krb5kdc[23337](info): set up 2 sockets
Jun 20 06:12:05 dist-test-slave-zt01 krb5kdc[23337](info): commencing operation
krb5kdc: starting...
W20260620 06:12:06.199718 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:12:06.321946 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:38257
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:06.424401 23350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:06.424613 23350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:06.424652 23350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:06.428781 23350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:06.428833 23350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:06.428853 23350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:06.428869 23350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:06.428884 23350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:06.433934 23350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:38257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:06.434816 23350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:06.435771 23350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:06.443480 23358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:06.443616 23350 server_base.cc:1034] running on GCE node
W20260620 06:12:06.443475 23355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:06.443478 23356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:06.444368 23350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:06.445348 23350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:06.446548 23350 hybrid_clock.cc:648] HybridClock initialized: now 1781935926446527 us; error 33 us; skew 500 ppm
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:06.449245 23350 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:12:06.450040 23350 webserver.cc:460] Webserver started at http://127.14.209.190:36285/ using document root <none> and password file <none>
I20260620 06:12:06.450556 23350 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:06.450641 23350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:06.450853 23350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:06.452783 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "806c009c87264d4bb4b0355a9e43798b"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "827ed3861560c32d91159d447e87a62b"
server_key_iv: "a2a6c312344f9991853e4e0209176960"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.453305 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "806c009c87264d4bb4b0355a9e43798b"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "827ed3861560c32d91159d447e87a62b"
server_key_iv: "a2a6c312344f9991853e4e0209176960"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.457083 23350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:12:06.459342 23365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.460506 23350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:12:06.460639 23350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "806c009c87264d4bb4b0355a9e43798b"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "827ed3861560c32d91159d447e87a62b"
server_key_iv: "a2a6c312344f9991853e4e0209176960"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.460738 23350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:06.472760 23350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:06.475854 23350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:06.476087 23350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:06.482707 23350 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:38257
I20260620 06:12:06.482784 23417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:38257 every 8 connection(s)
I20260620 06:12:06.483637 23350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:12:06.486279 23418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:06.486966 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23350
I20260620 06:12:06.487143 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:12:06.487447 15174 external_mini_cluster.cc:1428] Setting key a854f9ac3f4ae907bb3fb76e54ad8c01
I20260620 06:12:06.492036 23418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Bootstrap starting.
I20260620 06:12:06.494455 23418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:06.495388 23418 log.cc:824] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:12:06.497129 23418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: No bootstrap required, opened a new log
I20260620 06:12:06.500409 23418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:06.500626 23418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:06.500703 23418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806c009c87264d4bb4b0355a9e43798b, State: Initialized, Role: FOLLOWER
I20260620 06:12:06.501077 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:06.488786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.14.209.190:38257)
0620 06:12:06.489176 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:06.489192 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:06.489224 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:12:06.489810 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:06.489825 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:06.490124 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:06.490374 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:06.491276 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.492254 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.493385 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.493825 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.493939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:06.496667 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:06.496672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:06.496685 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:06.496709 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:06.498381 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:06.498827 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:06.498832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:06.498838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:06.498943 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:06.499246 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:06.499251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:06.499253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:06.499599 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:06.499808 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:06.500065 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 06:12:06.500317 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260620 06:12:06.501180 23418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:06.501330 23418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:12:06.501395 23418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:12:06.501489 23418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:06.502553 23418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:06.503132 23418 leader_election.cc:302] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 806c009c87264d4bb4b0355a9e43798b; no voters:
I20260620 06:12:06.503449 23418 leader_election.cc:288] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:12:06.503798 23423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:06.504088 23423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 LEADER]: Becoming Leader. State: Replica: 806c009c87264d4bb4b0355a9e43798b, State: Running, Role: LEADER
I20260620 06:12:06.504400 23423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:06.504602 23418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:12:06.505816 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806c009c87264d4bb4b0355a9e43798b. Latest consensus state: current_term: 1 leader_uuid: "806c009c87264d4bb4b0355a9e43798b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } } }
I20260620 06:12:06.505955 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:06.506284 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "806c009c87264d4bb4b0355a9e43798b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } } }
I20260620 06:12:06.506362 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: This master's current role is: LEADER
W20260620 06:12:06.509187 23438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:12:06.509250 23438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:12:06.509305 23435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:06.510006 23435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:06.514884 23435 catalog_manager.cc:1347] Generated new cluster ID: f069f581ff7942fea925d363e71ad73f
I20260620 06:12:06.514957 23435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:06.535262 23435 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:12:06.536128 23435 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:06.541689 23435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Generated new TSK 0
I20260620 06:12:06.542366 23435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:12:06.570326 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38257
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:06.674553 23444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:06.674772 23444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:06.674811 23444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:06.678684 23444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:06.678733 23444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:06.678822 23444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:12:06.683789 23444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:06.684649 23444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:06.685542 23444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:06.694288 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:06.694263 23449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:06.694293 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:06.694598 23444 server_base.cc:1034] running on GCE node
I20260620 06:12:06.695127 23444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:06.695747 23444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:06.696934 23444 hybrid_clock.cc:648] HybridClock initialized: now 1781935926696918 us; error 33 us; skew 500 ppm
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:06.699519 23444 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:12:06.700304 23444 webserver.cc:460] Webserver started at http://127.14.209.129:46113/ using document root <none> and password file <none>
I20260620 06:12:06.700810 23444 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:06.700896 23444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:06.701128 23444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:06.702936 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "160cface801d4666b794d1aea1da10e5"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "a5e9d6874aca4714be95b9a1a9b4a31b"
server_key_iv: "c4a53f6e1a9a0fd3e880972e8c815542"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.703532 23444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "160cface801d4666b794d1aea1da10e5"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "a5e9d6874aca4714be95b9a1a9b4a31b"
server_key_iv: "c4a53f6e1a9a0fd3e880972e8c815542"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.707263 23444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:12:06.709764 23459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.711344 23444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:12:06.711480 23444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "160cface801d4666b794d1aea1da10e5"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "a5e9d6874aca4714be95b9a1a9b4a31b"
server_key_iv: "c4a53f6e1a9a0fd3e880972e8c815542"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.711588 23444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:06.734584 23444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:06.737402 23444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:06.737593 23444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:06.738144 23444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:06.738893 23444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:06.738961 23444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.739042 23444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:06.739115 23444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.748366 23444 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:36907
I20260620 06:12:06.748406 23572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:36907 every 8 connection(s)
I20260620 06:12:06.749181 23444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:12:06.755834 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23444
I20260620 06:12:06.755975 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:12:06.756255 15174 external_mini_cluster.cc:1428] Setting key 8fc3fcad60e06d3e94bf938b839e8931
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:12:06.762171 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:06.750868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:49407 (local address 127.14.209.190:38257)
0620 06:12:06.750989 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:06.750993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:06.752051 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 06:12:06.752920 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:06.752924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:06.752980 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:06.753085 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:06.754146 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.754751 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.755852 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.756181 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.756294 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:06.758577 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:06.758581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:06.758584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:06.758609 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:06.760076 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:06.760648 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:06.760651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:06.760652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:06.760722 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:06.761103 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:06.761106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:06.761107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:06.761276 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:06.761351 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:06.761953 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 06:12:06.762061 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:12:06.763357 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:06.751277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:49407)
0620 06:12:06.751908 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:06.751943 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:06.752724 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:06.753247 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:06.753254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:06.753575 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:06.753998 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:06.754010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.754861 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.754865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:06.755720 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:06.755729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.756323 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.756327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:06.756490 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:06.757004 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:06.757018 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:06.758426 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:06.760189 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:06.760197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:06.760212 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:06.760553 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:06.760810 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:06.760816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:06.760821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:06.760999 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:06.761388 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:06.761397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:06.761697 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 06:12:06.761911 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 06:12:06.762248 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":116,"threads_started":1}
I20260620 06:12:06.764513 23573 heartbeater.cc:347] Connected to a master server at 127.14.209.190:38257
I20260620 06:12:06.764742 23573 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:06.765210 23573 heartbeater.cc:511] Master 127.14.209.190:38257 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:12:06.766683 23382 ts_manager.cc:194] Registered new tserver with Master: 160cface801d4666b794d1aea1da10e5 (127.14.209.129:36907)
I20260620 06:12:06.767756 23382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:49407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:12:06.781812 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38257
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:06.886706 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:06.886931 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:06.886970 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:06.890882 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:06.890935 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:06.890990 23578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:12:06.896061 23578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:06.896987 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:06.897927 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:06.907655 23586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:06.907634 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:06.907652 23584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:06.908382 23578 server_base.cc:1034] running on GCE node
I20260620 06:12:06.908767 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:06.909380 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:06.910600 23578 hybrid_clock.cc:648] HybridClock initialized: now 1781935926910571 us; error 43 us; skew 500 ppm
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:06.913259 23578 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:12:06.914048 23578 webserver.cc:460] Webserver started at http://127.14.209.130:34393/ using document root <none> and password file <none>
I20260620 06:12:06.914574 23578 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:06.914660 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:06.914877 23578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:06.916939 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "42b57c4c047f4e079e48dd3b68fee9a2"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "12273884e449bb1e8b17ca28e1845f4b"
server_key_iv: "fd79cd3a5259cbb8466ba012b7923e11"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.917474 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "42b57c4c047f4e079e48dd3b68fee9a2"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "12273884e449bb1e8b17ca28e1845f4b"
server_key_iv: "fd79cd3a5259cbb8466ba012b7923e11"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.920884 23578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:12:06.923209 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.924332 23578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:06.924469 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "42b57c4c047f4e079e48dd3b68fee9a2"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "12273884e449bb1e8b17ca28e1845f4b"
server_key_iv: "fd79cd3a5259cbb8466ba012b7923e11"
server_key_version: "encryptionkey@0"
I20260620 06:12:06.924578 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:06.941017 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:06.944073 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:06.944278 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:06.944857 23578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:06.945611 23578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:06.945679 23578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.945747 23578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:06.945801 23578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:06.954707 23578 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:32979
I20260620 06:12:06.954727 23706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:32979 every 8 connection(s)
I20260620 06:12:06.955564 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:12:06.957175 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23578
I20260620 06:12:06.957309 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:12:06.957608 15174 external_mini_cluster.cc:1428] Setting key 380d12aece639134a13de002cbae7561
Jun 20 06:12:06 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:12:06.969671 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:06.957259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:55343 (local address 127.14.209.190:38257)
0620 06:12:06.957401 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:06.957406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:06.958659 (+ 1253us) server_negotiation.cc:407] Connection header received
0620 06:12:06.959666 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:06.959669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:06.959719 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:06.959824 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:06.960905 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.961423 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.963232 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.963461 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.963550 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:06.966259 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:06.966263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:06.966265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:06.966287 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:06.967769 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:06.968263 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:06.968268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:06.968270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:06.968367 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:06.968694 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:06.968696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:06.968697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:06.968870 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:06.968946 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:06.969454 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 06:12:06.969549 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:12:06.970435 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:06.957700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:55343)
0620 06:12:06.958500 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:06.958550 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:06.959464 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:06.959987 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:06.959995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:06.960311 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:06.960769 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:06.960781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.961556 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.961583 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:06.963092 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:06.963106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:06.963619 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:06.963624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:06.963840 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:06.964468 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:06.964502 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:06.966089 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:06.967885 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:06.967891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:06.967904 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:06.968159 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:06.968475 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:06.968477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:06.968478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:06.968605 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:06.968971 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:06.968976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:06.969191 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 06:12:06.969363 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 06:12:06.969586 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":142,"threads_started":1}
I20260620 06:12:06.971745 23707 heartbeater.cc:347] Connected to a master server at 127.14.209.190:38257
I20260620 06:12:06.971971 23707 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:06.972424 23707 heartbeater.cc:511] Master 127.14.209.190:38257 requested a full tablet report, sending...
I20260620 06:12:06.973467 23382 ts_manager.cc:194] Registered new tserver with Master: 42b57c4c047f4e079e48dd3b68fee9a2 (127.14.209.130:32979)
I20260620 06:12:06.974310 23382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:55343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:12:06.987267 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38257
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:07.091972 23712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:07.092193 23712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:07.092232 23712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:07.096215 23712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:07.096264 23712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:07.096359 23712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:12:07.101534 23712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:07.102535 23712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:07.103560 23712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:12:07.112684 23712 server_base.cc:1034] running on GCE node
W20260620 06:12:07.112637 23717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.112632 23718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.112632 23720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.113261 23712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:07.113852 23712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:07.115028 23712 hybrid_clock.cc:648] HybridClock initialized: now 1781935927115018 us; error 45 us; skew 500 ppm
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:07.117650 23712 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:12:07.118353 23712 webserver.cc:460] Webserver started at http://127.14.209.131:44029/ using document root <none> and password file <none>
I20260620 06:12:07.118819 23712 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:07.118872 23712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:07.119019 23712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:07.120867 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "7069a063f7f54373932484d8b624e2e4"
format_stamp: "Formatted at 2026-06-20 06:12:07 on dist-test-slave-zt01"
server_key: "6ac4cd5a8432526c173b8416e3a051f2"
server_key_iv: "5ba0dc302c8507b2027374d9d1ef33d5"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.121349 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "7069a063f7f54373932484d8b624e2e4"
format_stamp: "Formatted at 2026-06-20 06:12:07 on dist-test-slave-zt01"
server_key: "6ac4cd5a8432526c173b8416e3a051f2"
server_key_iv: "5ba0dc302c8507b2027374d9d1ef33d5"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.124653 23712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:12:07.126747 23727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.128086 23712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:07.128228 23712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "7069a063f7f54373932484d8b624e2e4"
format_stamp: "Formatted at 2026-06-20 06:12:07 on dist-test-slave-zt01"
server_key: "6ac4cd5a8432526c173b8416e3a051f2"
server_key_iv: "5ba0dc302c8507b2027374d9d1ef33d5"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.128326 23712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:07.151511 23712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:07.154551 23712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:07.154700 23712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:07.155282 23712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:07.156026 23712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:07.156070 23712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.156108 23712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:07.156132 23712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.165006 23712 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:35627
I20260620 06:12:07.165025 23840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:35627 every 8 connection(s)
I20260620 06:12:07.165850 23712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:12:07.172711 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23712
I20260620 06:12:07.172873 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:12:07.173130 15174 external_mini_cluster.cc:1428] Setting key 40eee770ae1878463d11ae3cc98a7bd8
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:12:07.179539 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.167580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:49633 (local address 127.14.209.190:38257)
0620 06:12:07.167754 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.167758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.168708 (+ 950us) server_negotiation.cc:407] Connection header received
0620 06:12:07.169538 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.169541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.169589 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.169669 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.170912 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.171506 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.173273 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.173556 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.173693 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.176235 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:07.176238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:07.176240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:07.176263 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:07.177627 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:07.178080 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:07.178083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:07.178084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:07.178160 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:07.178535 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:07.178537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:07.178538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:07.178702 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:07.178791 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:07.179319 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 06:12:07.179417 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 06:12:07.180317 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.167917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:49633)
0620 06:12:07.168566 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.168604 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.169324 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.169951 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.169961 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.170289 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.170737 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.170749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.171751 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.171756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.173051 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.173069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.173739 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.173744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.174006 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.174630 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.174646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.176085 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:07.177734 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:07.177738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:07.177748 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:07.177972 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:07.178310 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:07.178313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:07.178314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:07.178446 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:07.178805 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:07.178810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:07.179015 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 06:12:07.179249 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 06:12:07.179490 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":117,"threads_started":1}
I20260620 06:12:07.181479 23841 heartbeater.cc:347] Connected to a master server at 127.14.209.190:38257
I20260620 06:12:07.181717 23841 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:07.182192 23841 heartbeater.cc:511] Master 127.14.209.190:38257 requested a full tablet report, sending...
I20260620 06:12:07.183236 23382 ts_manager.cc:194] Registered new tserver with Master: 7069a063f7f54373932484d8b624e2e4 (127.14.209.131:35627)
I20260620 06:12:07.183776 23382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:49633
I20260620 06:12:07.186713 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab.
Entry for principal oryx/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab.
WARNING: no policy specified for oryx/127.14.209.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab.
Entry for principal oryx/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab.
WARNING: no policy specified for oryx/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab.
Entry for principal oryx/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab.
I20260620 06:12:07.253443 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23444
I20260620 06:12:07.258960 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23578
I20260620 06:12:07.263948 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23712
I20260620 06:12:07.268980 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23350
I20260620 06:12:07.274781 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:38257
--webserver_interface=127.14.209.190
--webserver_port=36285
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:07.378695 23850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:07.378963 23850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:07.379065 23850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:07.382890 23850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:07.382973 23850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:07.383013 23850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:07.383080 23850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:07.383116 23850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:07.389322 23850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:38257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:07.390234 23850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:07.391161 23850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:07.398960 23855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.398979 23858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.399021 23856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.399178 23850 server_base.cc:1034] running on GCE node
I20260620 06:12:07.399763 23850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:07.400769 23850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:07.401985 23850 hybrid_clock.cc:648] HybridClock initialized: now 1781935927401960 us; error 49 us; skew 500 ppm
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:07.404642 23850 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:12:07.405371 23850 webserver.cc:460] Webserver started at http://127.14.209.190:36285/ using document root <none> and password file <none>
I20260620 06:12:07.405829 23850 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:07.405893 23850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:07.409315 23850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:12:07.411347 23865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.412495 23850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:12:07.412590 23850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "806c009c87264d4bb4b0355a9e43798b"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "827ed3861560c32d91159d447e87a62b"
server_key_iv: "a2a6c312344f9991853e4e0209176960"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.413049 23850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:07.428077 23850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:07.430956 23850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:07.431156 23850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:07.437283 23850 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:38257
I20260620 06:12:07.437347 23917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:38257 every 8 connection(s)
I20260620 06:12:07.438277 23850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:12:07.440140 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23850
I20260620 06:12:07.440610 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:36907
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:38257
--webserver_port=46113
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:12:07.442929 23918 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:12:07.444530 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Bootstrap starting.
I20260620 06:12:07.460096 23918 log.cc:824] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:07.467270 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:12:07.467634 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Bootstrap complete.
I20260620 06:12:07.471699 23918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:07.472057 23918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806c009c87264d4bb4b0355a9e43798b, State: Initialized, Role: FOLLOWER
I20260620 06:12:07.472527 23918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:07.472635 23918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:12:07.472721 23918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:12:07.472842 23918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:12:07.474658 23918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:07.475020 23918 leader_election.cc:302] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 806c009c87264d4bb4b0355a9e43798b; no voters:
I20260620 06:12:07.475384 23918 leader_election.cc:288] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:12:07.475517 23922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:12:07.475757 23922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [term 2 LEADER]: Becoming Leader. State: Replica: 806c009c87264d4bb4b0355a9e43798b, State: Running, Role: LEADER
I20260620 06:12:07.476104 23922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } }
I20260620 06:12:07.476490 23918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:12:07.477492 23923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "806c009c87264d4bb4b0355a9e43798b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } } }
I20260620 06:12:07.477625 23923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:07.477880 23924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806c009c87264d4bb4b0355a9e43798b. Latest consensus state: current_term: 2 leader_uuid: "806c009c87264d4bb4b0355a9e43798b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "806c009c87264d4bb4b0355a9e43798b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38257 } } }
I20260620 06:12:07.477954 23924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:07.477998 23931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:07.480897 23931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:07.481766 23931 catalog_manager.cc:1259] Loaded cluster ID: f069f581ff7942fea925d363e71ad73f
I20260620 06:12:07.481827 23931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:07.484294 23931 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:07.484915 23931 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 806c009c87264d4bb4b0355a9e43798b: Loaded TSK: 0
I20260620 06:12:07.486029 23931 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:12:07.549887 23920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:07.550112 23920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:07.550151 23920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:07.554039 23920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:07.554085 23920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:07.554140 23920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:12:07.559122 23920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.129.keytab
--principal=oryx/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:36907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=46113
--tserver_master_addrs=127.14.209.190:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:07.559974 23920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:07.560869 23920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:07.569716 23944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.569787 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.569917 23920 server_base.cc:1034] running on GCE node
W20260620 06:12:07.569731 23945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.570498 23920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:07.571197 23920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:07.572396 23920 hybrid_clock.cc:648] HybridClock initialized: now 1781935927572364 us; error 39 us; skew 500 ppm
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935927, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:07.575829 23920 init.cc:377] Logged in from keytab as oryx/127.14.209.129@KRBTEST.COM (short username oryx)
I20260620 06:12:07.576803 23920 webserver.cc:460] Webserver started at http://127.14.209.129:46113/ using document root <none> and password file <none>
I20260620 06:12:07.577464 23920 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:07.577605 23920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:07.582283 23920 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:07.584712 23954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.586124 23920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:12:07.586239 23920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "160cface801d4666b794d1aea1da10e5"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "a5e9d6874aca4714be95b9a1a9b4a31b"
server_key_iv: "c4a53f6e1a9a0fd3e880972e8c815542"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.586882 23920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:07.599676 23920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:07.602610 23920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:07.602798 23920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:07.603446 23920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:07.604188 23920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:07.604233 23920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.604292 23920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:07.604323 23920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.614720 23920 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:36907
I20260620 06:12:07.614739 24067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:36907 every 8 connection(s)
I20260620 06:12:07.615885 23920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:12:07.616698 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 23920
I20260620 06:12:07.617161 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:32979
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:38257
--webserver_port=34393
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.631583 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.620288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:48181)
0620 06:12:07.620946 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.620993 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.621747 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.622987 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.622993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.623396 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.623827 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.623848 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.625791 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.625795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.626631 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.626640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.627757 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.627760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.627891 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.628492 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.628509 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.630844 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":127,"threads_started":1}
W20260620 06:12:07.631840 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.620107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:48181 (local address 127.14.209.190:38257)
0620 06:12:07.620598 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.620617 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.621110 (+ 493us) server_negotiation.cc:407] Connection header received
0620 06:12:07.622144 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.622162 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.622562 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.622931 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.624007 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.625653 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.626757 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.627586 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.627776 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.631103 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:48181: BlockingRecv error: recv got EOF from 127.14.209.129:48181 (error 108)
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":121,"threads_started":1}
W20260620 06:12:07.632014 24068 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:38257 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.209.190:38257: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.637542 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.632404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:54315)
0620 06:12:07.632524 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.632540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.632613 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.632868 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.632870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.633021 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.633103 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.633108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.634075 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.634078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.634586 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.634593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.635210 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.635213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.635278 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.635732 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.635752 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.637431 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 06:12:07.637809 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.632496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:54315 (local address 127.14.209.190:38257)
0620 06:12:07.632617 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.632621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.632635 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:12:07.632684 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.632687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.632740 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.632818 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.633208 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.633967 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.634698 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.635102 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.635204 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.637565 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:54315: BlockingRecv error: recv got EOF from 127.14.209.129:54315 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.643203 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.637927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:41691)
0620 06:12:07.638228 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.638245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.638326 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.638558 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.638560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.638703 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.638779 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.638783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.639798 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.639801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.640340 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.640348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.640903 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.640905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.640975 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.641448 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.641459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.643045 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
W20260620 06:12:07.643293 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.638115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:41691 (local address 127.14.209.190:38257)
0620 06:12:07.638263 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.638267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.638302 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:12:07.638407 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.638412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.638452 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.638531 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.638877 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.639685 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.640454 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.640783 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.640885 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.643197 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:41691: BlockingRecv error: recv got EOF from 127.14.209.129:41691 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 06:12:07.643384 24068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:12:07.726641 24070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:07.726902 24070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:07.726967 24070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:07.730973 24070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:07.731043 24070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:07.731168 24070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:12:07.736241 24070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.130.keytab
--principal=oryx/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:32979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=34393
--tserver_master_addrs=127.14.209.190:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:07.737170 24070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:07.738093 24070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:07.746731 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.746747 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.747155 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.747197 24070 server_base.cc:1034] running on GCE node
I20260620 06:12:07.747563 24070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:07.748148 24070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:07.749332 24070 hybrid_clock.cc:648] HybridClock initialized: now 1781935927749314 us; error 29 us; skew 500 ppm
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935927, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:07.751879 24070 init.cc:377] Logged in from keytab as oryx/127.14.209.130@KRBTEST.COM (short username oryx)
I20260620 06:12:07.752763 24070 webserver.cc:460] Webserver started at http://127.14.209.130:34393/ using document root <none> and password file <none>
I20260620 06:12:07.753276 24070 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:07.753381 24070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:07.756927 24070 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260620 06:12:07.758790 24088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.760044 24070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:07.760169 24070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "42b57c4c047f4e079e48dd3b68fee9a2"
format_stamp: "Formatted at 2026-06-20 06:12:06 on dist-test-slave-zt01"
server_key: "12273884e449bb1e8b17ca28e1845f4b"
server_key_iv: "fd79cd3a5259cbb8466ba012b7923e11"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.760660 24070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:07.778043 24070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:07.780824 24070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:07.781018 24070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:07.781535 24070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:07.782254 24070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:07.782325 24070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.782388 24070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:07.782437 24070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.791568 24070 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:32979
I20260620 06:12:07.791597 24201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:32979 every 8 connection(s)
I20260620 06:12:07.792523 24070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:12:07.793437 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24070
I20260620 06:12:07.793905 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:35627
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:38257
--webserver_port=44029
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:12:07.804773 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.795711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:59217 (local address 127.14.209.190:38257)
0620 06:12:07.795807 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.795812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.795834 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:12:07.796869 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.796875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.796943 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.797039 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.798099 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.799025 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.800775 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.801010 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.801128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.804654 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:59217: BlockingRecv error: recv got EOF from 127.14.209.130:59217 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 06:12:07.805330 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.794657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:59217)
0620 06:12:07.795438 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.795495 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.796624 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.797177 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.797183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.797507 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.797953 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.797966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.799182 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.799185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.800660 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.800674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.801160 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.801165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.801362 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.802059 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.802080 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.804542 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":152,"threads_started":1}
W20260620 06:12:07.805763 24202 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:38257 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.209.190:38257: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.812140 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.806195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:36407)
0620 06:12:07.806477 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.806493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.806578 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.806823 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.806826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.806986 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.807128 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.807135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.807848 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.807852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.808388 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.808396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.808850 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.808854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.808942 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.809455 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.809474 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.811996 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210}
W20260620 06:12:07.812230 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.806335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:36407 (local address 127.14.209.190:38257)
0620 06:12:07.806462 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.806466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.806540 (+ 74us) server_negotiation.cc:407] Connection header received
0620 06:12:07.806673 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.806676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.806723 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.806810 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.807246 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.807732 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.808511 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.808740 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.808849 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.812112 (+ 3263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:36407: BlockingRecv error: recv got EOF from 127.14.209.130:36407 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.817997 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.812837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:56459)
0620 06:12:07.812980 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.812993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.813057 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.813266 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.813269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.813424 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.813538 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.813544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.814271 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.814275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.815174 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.815188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.815618 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.815622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.815700 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.816182 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.816193 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.817872 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 06:12:07.818110 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.812907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:56459 (local address 127.14.209.190:38257)
0620 06:12:07.813028 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.813031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.813043 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:12:07.813134 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.813136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.813167 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.813232 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.813680 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.814166 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.815298 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.815500 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.815594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.817976 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:56459: BlockingRecv error: recv got EOF from 127.14.209.130:56459 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 06:12:07.818356 24202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:12:07.905015 24205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:07.905239 24205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:07.905280 24205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:07.909361 24205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:07.909415 24205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:07.909524 24205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:12:07.914661 24205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/krb5kdc/oryx_127.14.209.131.keytab
--principal=oryx/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:35627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=44029
--tserver_master_addrs=127.14.209.190:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:07.915696 24205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:07.916697 24205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:07.925865 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:07.925904 24213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.926086 24205 server_base.cc:1034] running on GCE node
W20260620 06:12:07.926259 24211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:07.926635 24205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:07.927265 24205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:07.928426 24205 hybrid_clock.cc:648] HybridClock initialized: now 1781935927928411 us; error 26 us; skew 500 ppm
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935927, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:07.931226 24205 init.cc:377] Logged in from keytab as oryx/127.14.209.131@KRBTEST.COM (short username oryx)
I20260620 06:12:07.932051 24205 webserver.cc:460] Webserver started at http://127.14.209.131:44029/ using document root <none> and password file <none>
I20260620 06:12:07.932520 24205 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:07.932590 24205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:07.936113 24205 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:12:07.937983 24220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.939316 24205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:12:07.939453 24205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "7069a063f7f54373932484d8b624e2e4"
format_stamp: "Formatted at 2026-06-20 06:12:07 on dist-test-slave-zt01"
server_key: "6ac4cd5a8432526c173b8416e3a051f2"
server_key_iv: "5ba0dc302c8507b2027374d9d1ef33d5"
server_key_version: "encryptionkey@0"
I20260620 06:12:07.939963 24205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:07.954480 24205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:07.958446 24205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:07.958606 24205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:07.959156 24205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:07.959911 24205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:07.959959 24205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.959998 24205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:07.960023 24205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:07.968333 24205 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:35627
I20260620 06:12:07.968397 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:35627 every 8 connection(s)
I20260620 06:12:07.969365 24205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:12:07.970597 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24205
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.980368 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.971580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.14.209.190:38257)
0620 06:12:07.971826 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.971830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.971850 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:12:07.971888 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.971890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.971929 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.972031 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.972807 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.973348 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.974190 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.974423 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.974536 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.976224 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:07.976229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:07.976241 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:07.976267 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:07.978286 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:07.978777 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:07.978783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:07.978791 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:07.978930 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:07.979235 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:07.979239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:07.979241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:07.979568 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:07.979771 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:07.980006 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 06:12:07.980242 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260620 06:12:07.981400 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.971566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:39709)
0620 06:12:07.972271 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.972318 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.973060 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.973526 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.973533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.973841 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.974314 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.974326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.975224 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.975228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.976483 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.976492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.977106 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.977111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.977296 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.977873 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.977914 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.980524 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":152,"threads_started":1}
W20260620 06:12:07.981936 24334 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:38257 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.209.190:38257: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
W20260620 06:12:07.982338 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.971097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:39709 (local address 127.14.209.190:38257)
0620 06:12:07.971258 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.971263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.972424 (+ 1161us) server_negotiation.cc:407] Connection header received
0620 06:12:07.973252 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.973255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.973304 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.973376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.974447 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.975092 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.976652 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.976936 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.977041 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.982203 (+ 5162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:39709: BlockingRecv error: recv got EOF from 127.14.209.131:39709 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.987154 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.982365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:52591)
0620 06:12:07.982532 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.982549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.982626 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.983022 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.983025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.983227 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.983330 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.983336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.984092 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.984094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.984607 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.984615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.984992 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.984995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.985059 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.985426 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.985436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.987011 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 06:12:07.987265 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.982444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:52591 (local address 127.14.209.190:38257)
0620 06:12:07.982845 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.982848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.982858 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:12:07.982883 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.982885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.982913 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.983009 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.983437 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.983997 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.984704 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.984898 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.984971 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.987167 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:52591: BlockingRecv error: recv got EOF from 127.14.209.131:52591 (error 108)
Metrics: {"server-negotiator.queue_time_us":346}
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:07 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:07.991974 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:07.987546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:54619)
0620 06:12:07.987790 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:07.987805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:07.987887 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:07.988059 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:07.988061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:07.988199 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:07.988275 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.988279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.988876 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.988879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.989443 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:07.989450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.989821 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.989824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:07.989896 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.990287 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:07.990298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:07.991868 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
W20260620 06:12:07.992097 24071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:07.987640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:54619 (local address 127.14.209.190:38257)
0620 06:12:07.987758 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:07.987761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:07.987875 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:12:07.987940 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:07.987942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:07.987972 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:07.988032 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:07.988359 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.988794 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.989535 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:07.989726 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:07.989794 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:07.992001 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:54619: BlockingRecv error: recv got EOF from 127.14.209.131:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 06:12:07.992259 24334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 06:12:08 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:08 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:08.649730 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:08.643958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:49185)
0620 06:12:08.644245 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:08.644258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:08.644377 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:08.644675 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:08.644678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:08.644873 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:08.645016 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:08.645023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.645822 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.645827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:08.646533 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:08.646543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.647100 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.647104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:08.647212 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:08.647698 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:08.647710 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:08.649562 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
W20260620 06:12:08.649819 24341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:08.644110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:49185 (local address 127.14.209.190:38257)
0620 06:12:08.644371 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:08.644375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:08.644388 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:08.644473 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:08.644476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:08.644515 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:08.644591 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:08.645152 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.645701 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.646661 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.646938 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.647061 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:08.649690 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:49185: BlockingRecv error: recv got EOF from 127.14.209.129:49185 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":115,"threads_started":1}
Jun 20 06:12:08 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:08 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:08.824469 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:08.818952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:60063)
0620 06:12:08.819263 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:08.819276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:08.819375 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:08.819635 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:08.819638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:08.819801 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:08.819908 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:08.819913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.820811 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.820814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:08.821680 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:08.821689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.822083 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.822086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:08.822154 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:08.822592 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:08.822602 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:08.824358 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":148,"threads_started":1}
W20260620 06:12:08.824594 24341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:08.819113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:60063 (local address 127.14.209.190:38257)
0620 06:12:08.819284 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:08.819287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:08.819347 (+ 60us) server_negotiation.cc:407] Connection header received
0620 06:12:08.819474 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:08.819476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:08.819520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:08.819591 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:08.820031 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.820676 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.821777 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.821991 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.822068 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:08.824480 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:60063: BlockingRecv error: recv got EOF from 127.14.209.130:60063 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 06:12:08 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:08 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:08.998565 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:08.992842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:52201)
0620 06:12:08.993101 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:08.993117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:08.993231 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:08.993531 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:08.993534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:08.993733 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:08.993820 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:08.993826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.994576 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.994582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:08.995272 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:08.995282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.995742 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.995746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:08.995842 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:08.996463 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:08.996476 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:08.998379 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
W20260620 06:12:08.998600 24341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:08.992882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:52201 (local address 127.14.209.190:38257)
0620 06:12:08.993050 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:08.993054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:08.993176 (+ 122us) server_negotiation.cc:407] Connection header received
0620 06:12:08.993346 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:08.993349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:08.993420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:08.993509 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:08.993941 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.994465 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.995390 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:08.995632 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:08.995742 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:08.998500 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:52201: BlockingRecv error: recv got EOF from 127.14.209.131:52201 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 06:12:09 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:09 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:09.655529 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:09.650453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:43923)
0620 06:12:09.650749 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:09.650763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:09.650847 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:09.651131 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:09.651133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:09.651296 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:09.651387 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:09.651392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.652101 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.652103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:09.652629 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:09.652635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.653077 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.653079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:09.653135 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:09.653689 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:09.653701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:09.655393 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
W20260620 06:12:09.655617 24345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:09.650569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:43923 (local address 127.14.209.190:38257)
0620 06:12:09.650786 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:09.650789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:09.650805 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:12:09.650933 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:09.650935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:09.650981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:09.651113 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:09.651491 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.652004 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.652748 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.652968 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.653054 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:09.655520 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:43923: BlockingRecv error: recv got EOF from 127.14.209.129:43923 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":55,"threads_started":1}
Jun 20 06:12:09 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:09 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:09.830586 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:09.825313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:37805)
0620 06:12:09.825564 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:09.825578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:09.825670 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:09.825893 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:09.825895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:09.826076 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:09.826170 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:09.826176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.826854 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.826856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:09.827436 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:09.827445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.827863 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.827867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:09.827954 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:09.828494 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:09.828509 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:09.830468 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
W20260620 06:12:09.830740 24345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:09.825387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:37805 (local address 127.14.209.190:38257)
0620 06:12:09.825535 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:09.825538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:09.825643 (+ 105us) server_negotiation.cc:407] Connection header received
0620 06:12:09.825745 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:09.825747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:09.825789 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:09.825860 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:09.826266 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.826768 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.827530 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:09.827744 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:09.827821 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:09.830592 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:37805: BlockingRecv error: recv got EOF from 127.14.209.130:37805 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 06:12:09 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:09 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:10.004621 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:09.999330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:57625)
0620 06:12:09.999621 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:09.999635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:09.999778 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:10.000027 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:10.000029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:10.000191 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:10.000282 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:10.000288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.000987 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.000990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:10.001530 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:10.001537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.001973 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.001975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:10.002034 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:10.002491 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:10.002502 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:10.004503 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
W20260620 06:12:10.004833 24345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:09.999422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:57625 (local address 127.14.209.190:38257)
0620 06:12:09.999561 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:09.999565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:09.999775 (+ 210us) server_negotiation.cc:407] Connection header received
0620 06:12:09.999872 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:09.999877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:09.999929 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:09.999994 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:10.000383 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.000896 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.001667 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.001879 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.001956 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:10.004690 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:57625: BlockingRecv error: recv got EOF from 127.14.209.131:57625 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 06:12:10 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:10 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:10.661298 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:10.656319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:44047)
0620 06:12:10.656608 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:10.656621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:10.656698 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:10.656940 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:10.656942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:10.657112 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:10.657204 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:10.657209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.657920 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.657923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:10.658461 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:10.658468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.658879 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.658881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:10.658940 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:10.659462 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:10.659473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:10.661184 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
W20260620 06:12:10.661473 24349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:10.656387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:44047 (local address 127.14.209.190:38257)
0620 06:12:10.656641 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:10.656644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:10.656672 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:12:10.656793 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:10.656795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:10.656836 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:10.656907 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:10.657313 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.657829 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.658561 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.658787 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.658865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:10.661314 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:44047: BlockingRecv error: recv got EOF from 127.14.209.129:44047 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
Jun 20 06:12:10 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:10 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:10.837152 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:10.831384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:34823)
0620 06:12:10.831647 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:10.831661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:10.831804 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:10.832084 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:10.832087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:10.832282 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:10.832404 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:10.832413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.833468 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.833473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:10.834091 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:10.834099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.834522 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.834525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:10.834591 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:10.835101 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:10.835119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:10.837021 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
W20260620 06:12:10.837271 24349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:10.831435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:34823 (local address 127.14.209.190:38257)
0620 06:12:10.831592 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:10.831598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:10.831738 (+ 140us) server_negotiation.cc:407] Connection header received
0620 06:12:10.831914 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:10.831917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:10.831973 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:10.832061 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:10.832555 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.833355 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.834195 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:10.834407 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:10.834486 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:10.837154 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:34823: BlockingRecv error: recv got EOF from 127.14.209.130:34823 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 06:12:11 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:11 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:11.010689 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:11.005322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:48895)
0620 06:12:11.005562 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:11.005575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:11.005678 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:11.005899 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:11.005902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:11.006067 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:11.006163 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:11.006168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.006887 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.006890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:11.007462 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:11.007471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.007903 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.007906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:11.007980 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:11.008458 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:11.008472 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:11.010509 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
W20260620 06:12:11.010731 24349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:11.005358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:48895 (local address 127.14.209.190:38257)
0620 06:12:11.005509 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:11.005512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:11.005641 (+ 129us) server_negotiation.cc:407] Connection header received
0620 06:12:11.005756 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:11.005758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:11.005800 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:11.005867 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:11.006258 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.006787 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.007586 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.007805 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.007891 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:11.010632 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:48895: BlockingRecv error: recv got EOF from 127.14.209.131:48895 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 06:12:11 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:11 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:11.670477 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:11.662052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:36229)
0620 06:12:11.662309 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:11.662323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:11.662412 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:11.662667 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:11.662669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:11.662844 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:11.662934 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:11.662940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.663710 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.663713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:11.664282 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:11.664290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.664734 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.664736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:11.664811 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:11.668410 (+ 3599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:11.668422 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:11.670366 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
W20260620 06:12:11.670616 24353 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:11.662109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:36229 (local address 127.14.209.190:38257)
0620 06:12:11.662351 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:11.662355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:11.662368 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:11.662491 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:11.662493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:11.662536 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:11.662633 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:11.663038 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.663617 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.664377 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.664603 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.664725 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:11.670486 (+ 5761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:36229: BlockingRecv error: recv got EOF from 127.14.209.129:36229 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":58,"threads_started":1}
Jun 20 06:12:11 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:11 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:11.843454 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:11.838001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:53127)
0620 06:12:11.838297 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:11.838311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:11.838410 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:11.838646 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:11.838647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:11.838872 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:11.839029 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:11.839037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.839826 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.839831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:11.840386 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:11.840393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.840865 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.840869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:11.840952 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:11.841414 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:11.841424 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:11.843279 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":141,"threads_started":1}
W20260620 06:12:11.843493 24353 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:11.838087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:53127 (local address 127.14.209.190:38257)
0620 06:12:11.838247 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:11.838254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:11.838383 (+ 129us) server_negotiation.cc:407] Connection header received
0620 06:12:11.838475 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:11.838478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:11.838520 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:11.838641 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:11.839179 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.839713 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.840503 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:11.840754 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:11.840864 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:11.843405 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:53127: BlockingRecv error: recv got EOF from 127.14.209.130:53127 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 06:12:12 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:12 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:12.016991 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:12.011580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:35221)
0620 06:12:12.011844 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:12.011858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:12.011966 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:12.012200 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:12.012202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:12.012384 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:12.012486 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:12.012493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.013284 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.013287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:12.013871 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:12.013879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.014322 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.014324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:12.014384 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:12.014927 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:12.014944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:12.016856 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
W20260620 06:12:12.017325 24353 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:12.011624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:35221 (local address 127.14.209.190:38257)
0620 06:12:12.011772 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:12.011775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:12.012000 (+ 225us) server_negotiation.cc:407] Connection header received
0620 06:12:12.012043 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:12.012045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:12.012089 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:12.012161 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:12.012604 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.013167 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.014000 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.014225 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.014311 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:12.017003 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:35221: BlockingRecv error: recv got EOF from 127.14.209.131:35221 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 06:12:12 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:12 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:12.676684 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:12.671404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:34651)
0620 06:12:12.671678 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:12.671691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:12.671769 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:12.671982 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:12.671984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:12.672153 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:12.672245 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:12.672249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.672988 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.672990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:12.673544 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:12.673551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.673960 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.673962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:12.674022 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:12.674554 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:12.674566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:12.676504 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":109,"threads_started":1}
W20260620 06:12:12.676730 24357 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:12.671457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:34651 (local address 127.14.209.190:38257)
0620 06:12:12.671719 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:12.671723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:12.671736 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:12.671853 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:12.671855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:12.671894 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:12.671970 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:12.672358 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.672890 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.673646 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.673860 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.673948 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:12.676616 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:34651: BlockingRecv error: recv got EOF from 127.14.209.129:34651 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
Jun 20 06:12:12 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:12 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:12.849697 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:12.844208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:35067)
0620 06:12:12.844623 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:12.844639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:12.844776 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:12.845051 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:12.845054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:12.845255 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:12.845343 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:12.845349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.846105 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.846110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:12.846679 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:12.846687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.847192 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.847196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:12.847283 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:12.847793 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:12.847804 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:12.849531 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":98,"threads_started":1}
W20260620 06:12:12.849768 24357 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:12.844385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:35067 (local address 127.14.209.190:38257)
0620 06:12:12.844623 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:12.844630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:12.844723 (+ 93us) server_negotiation.cc:407] Connection header received
0620 06:12:12.844877 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:12.844882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:12.844937 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:12.845027 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:12.845467 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.845995 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.846804 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:12.847036 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:12.847194 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:12.849663 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:35067: BlockingRecv error: recv got EOF from 127.14.209.130:35067 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Jun 20 06:12:13 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:13 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:13.023775 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:13.017841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:35745)
0620 06:12:13.018164 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:13.018187 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:13.018323 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:13.018632 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:13.018635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:13.018858 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:13.018952 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:13.018958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.019793 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.019800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:13.020427 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:13.020436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.020941 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.020945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:13.021042 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:13.021626 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:13.021649 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:13.023610 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
W20260620 06:12:13.023837 24357 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:13.017940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:35745 (local address 127.14.209.190:38257)
0620 06:12:13.018109 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:13.018114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:13.018277 (+ 163us) server_negotiation.cc:407] Connection header received
0620 06:12:13.018445 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:13.018451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:13.018513 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:13.018610 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:13.019102 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.019674 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.020564 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.020811 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.020942 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:13.023731 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:35745: BlockingRecv error: recv got EOF from 127.14.209.131:35745 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 06:12:13 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:13 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:13.683135 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:13.677455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:40531)
0620 06:12:13.677759 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:13.677779 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:13.677910 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:13.678238 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:13.678242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:13.678502 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:13.678625 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:13.678633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.679459 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.679462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:13.680006 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:13.680013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.680539 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.680543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:13.680623 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:13.681134 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:13.681145 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:13.682962 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
W20260620 06:12:13.683362 24361 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:13.677518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:40531 (local address 127.14.209.190:38257)
0620 06:12:13.677823 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:13.677826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:13.677839 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:13.678012 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:13.678017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:13.678081 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:13.678199 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:13.678792 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.679367 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.680139 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.680397 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.680513 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:13.683209 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:40531: BlockingRecv error: recv got EOF from 127.14.209.129:40531 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
Jun 20 06:12:13 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:13 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:13.855929 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:13.850403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:35255)
0620 06:12:13.850687 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:13.850701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:13.850794 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:13.851121 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:13.851127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:13.851306 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:13.851401 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:13.851407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.852188 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.852191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:13.852762 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:13.852770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.853233 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.853235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:13.853298 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:13.853851 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:13.853863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:13.855764 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
W20260620 06:12:13.855997 24361 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:13.850515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:35255 (local address 127.14.209.190:38257)
0620 06:12:13.850653 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:13.850657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:13.850770 (+ 113us) server_negotiation.cc:407] Connection header received
0620 06:12:13.850872 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:13.850874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:13.850917 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:13.850995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:13.851538 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.852084 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.852875 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:13.853127 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:13.853209 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:13.855879 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:35255: BlockingRecv error: recv got EOF from 127.14.209.130:35255 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 06:12:14 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:14 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:14.030010 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:14.024467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:33489)
0620 06:12:14.024757 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:14.024770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:14.024860 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:14.025046 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:14.025078 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:14.025260 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:14.025352 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:14.025357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.026081 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.026083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:14.026673 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:14.026681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.027188 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.027191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:14.027252 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:14.027807 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:14.027820 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:14.029831 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
W20260620 06:12:14.030216 24361 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:14.024557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:33489 (local address 127.14.209.190:38257)
0620 06:12:14.024684 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:14.024687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:14.024835 (+ 148us) server_negotiation.cc:407] Connection header received
0620 06:12:14.024924 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:14.024926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:14.024969 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:14.025037 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:14.025446 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.025964 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.026794 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.027009 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.027118 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:14.030002 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:33489: BlockingRecv error: recv got EOF from 127.14.209.131:33489 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:12:14 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:14 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:14.689544 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:14.683842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:41693)
0620 06:12:14.684171 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:14.684185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:14.684292 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:14.684539 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:14.684541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:14.684741 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:14.684836 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:14.684841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.685815 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.685821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:14.686595 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:14.686612 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.687115 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.687119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:14.687199 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:14.687678 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:14.687689 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:14.689436 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
W20260620 06:12:14.689728 24365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:14.683945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:41693 (local address 127.14.209.190:38257)
0620 06:12:14.684206 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:14.684210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:14.684258 (+ 48us) server_negotiation.cc:407] Connection header received
0620 06:12:14.684377 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:14.684379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:14.684422 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:14.684496 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:14.684968 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.685708 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.686726 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.686945 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.687026 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:14.689555 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:41693: BlockingRecv error: recv got EOF from 127.14.209.129:41693 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
Jun 20 06:12:14 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:14 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:14.861790 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:14.856641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:51609)
0620 06:12:14.856948 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:14.856962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:14.857058 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:14.857267 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:14.857268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:14.857453 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:14.857546 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:14.857552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.858271 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.858274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:14.858855 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:14.858864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.859324 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.859326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:14.859390 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:14.859858 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:14.859869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:14.861681 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260620 06:12:14.861909 24365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:14.856723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:51609 (local address 127.14.209.190:38257)
0620 06:12:14.856885 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:14.856895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:14.857029 (+ 134us) server_negotiation.cc:407] Connection header received
0620 06:12:14.857130 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:14.857132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:14.857172 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:14.857237 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:14.857646 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.858162 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.858959 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:14.859232 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:14.859309 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:14.861799 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:51609: BlockingRecv error: recv got EOF from 127.14.209.130:51609 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 06:12:15 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:15 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:15.036062 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:15.030717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:46205)
0620 06:12:15.031021 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:15.031034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:15.031191 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:15.031422 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:15.031424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:15.031598 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:15.031703 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:15.031708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.032472 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.032473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:15.033015 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:15.033022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.033444 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.033446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:15.033510 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:15.034069 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:15.034082 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:15.035950 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
W20260620 06:12:15.036223 24365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:15.030821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:46205 (local address 127.14.209.190:38257)
0620 06:12:15.030981 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:15.030985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:15.031169 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:12:15.031268 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:15.031270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:15.031317 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:15.031388 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:15.031822 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.032368 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.033125 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.033338 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.033416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:15.036090 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:46205: BlockingRecv error: recv got EOF from 127.14.209.131:46205 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 06:12:15 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:15 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:15.695662 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:15.690318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:38077)
0620 06:12:15.690600 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:15.690613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:15.690696 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:15.690951 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:15.690953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:15.691188 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:15.691282 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:15.691288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.692187 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.692190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:15.692765 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:15.692773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.693191 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.693193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:15.693252 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:15.693752 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:15.693763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:15.695553 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
W20260620 06:12:15.695854 24369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:15.690472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:38077 (local address 127.14.209.190:38257)
0620 06:12:15.690679 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:15.690682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:15.690695 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:15.690797 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:15.690799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:15.690837 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:15.690944 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:15.691385 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.692078 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.692863 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.693096 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.693174 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:15.695674 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:38077: BlockingRecv error: recv got EOF from 127.14.209.129:38077 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
Jun 20 06:12:15 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:15 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:15.868330 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:15.862669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:54917)
0620 06:12:15.863020 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:15.863040 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:15.863188 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:15.863467 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:15.863470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:15.863680 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:15.863768 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:15.863773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.864706 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.864712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:15.865270 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:15.865278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.865732 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.865736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:15.865821 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:15.866345 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:15.866355 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:15.868221 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
W20260620 06:12:15.868466 24369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:15.862792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:54917 (local address 127.14.209.190:38257)
0620 06:12:15.862972 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:15.862977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:15.863138 (+ 161us) server_negotiation.cc:407] Connection header received
0620 06:12:15.863317 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:15.863319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:15.863363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:15.863447 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:15.863901 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.864591 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.865393 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:15.865618 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:15.865731 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:15.868353 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:54917: BlockingRecv error: recv got EOF from 127.14.209.130:54917 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 06:12:16 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:16 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:16.042286 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:16.036820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:34471)
0620 06:12:16.037133 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:16.037146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:16.037243 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:16.037477 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:16.037479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:16.037646 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:16.037738 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:16.037744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.038452 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.038454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:16.039001 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:16.039009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.039508 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.039511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:16.039577 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:16.040066 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:16.040077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:16.042110 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
W20260620 06:12:16.042392 24369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:16.036955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:34471 (local address 127.14.209.190:38257)
0620 06:12:16.037123 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:16.037126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:16.037197 (+ 71us) server_negotiation.cc:407] Connection header received
0620 06:12:16.037327 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:16.037329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:16.037373 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:16.037438 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:16.037833 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.038357 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.039180 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.039388 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.039467 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:16.042268 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:34471: BlockingRecv error: recv got EOF from 127.14.209.131:34471 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 06:12:16 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:16 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:16.704983 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:16.696373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:34653)
0620 06:12:16.696647 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:16.696660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:16.696752 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:16.697041 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:16.697044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:16.697218 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:16.697322 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:16.697327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.698132 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.698135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:16.698803 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:16.698812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.699349 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.699353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:16.699440 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:16.702955 (+ 3515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:16.702967 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:16.704878 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260620 06:12:16.705178 24373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:16.696422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:34653 (local address 127.14.209.190:38257)
0620 06:12:16.696657 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:16.696660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:16.696723 (+ 63us) server_negotiation.cc:407] Connection header received
0620 06:12:16.696830 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:16.696832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:16.696884 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:16.696985 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:16.697456 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.698023 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.698934 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.699233 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.699347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:16.705042 (+ 5695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:34653: BlockingRecv error: recv got EOF from 127.14.209.129:34653 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":57,"threads_started":1}
Jun 20 06:12:16 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:16 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:16.874567 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:16.869118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:53745)
0620 06:12:16.869366 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:16.869379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:16.869474 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:16.869708 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:16.869710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:16.869878 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:16.870058 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:16.870073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.870984 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.870989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:16.871659 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:16.871668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.872102 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.872104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:16.872163 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:16.872613 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:16.872624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:16.874457 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
W20260620 06:12:16.874743 24373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:16.869227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:53745 (local address 127.14.209.190:38257)
0620 06:12:16.869362 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:16.869366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:16.869446 (+ 80us) server_negotiation.cc:407] Connection header received
0620 06:12:16.869561 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:16.869563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:16.869604 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:16.869675 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:16.870198 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.870900 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.871769 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:16.872000 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:16.872082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:16.874598 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:53745: BlockingRecv error: recv got EOF from 127.14.209.130:53745 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 06:12:17 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:17 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:17.048458 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:17.043138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:34137)
0620 06:12:17.043450 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:17.043466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:17.043558 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:17.043768 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:17.043770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:17.044013 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:17.044107 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:17.044113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.044906 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.044909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:17.045504 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:17.045512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.045979 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.045982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:17.046041 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:17.046513 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:17.046525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:17.048339 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":80,"threads_started":1}
W20260620 06:12:17.048591 24373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:17.043145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:34137 (local address 127.14.209.190:38257)
0620 06:12:17.043292 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:17.043297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:17.043534 (+ 237us) server_negotiation.cc:407] Connection header received
0620 06:12:17.043629 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:17.043631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:17.043675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:17.043741 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:17.044226 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.044807 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.045623 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.045849 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.045941 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:17.048473 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:34137: BlockingRecv error: recv got EOF from 127.14.209.131:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:12:17 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:17 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:17.711335 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:17.705821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:50683)
0620 06:12:17.706084 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:17.706102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:17.706208 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:17.706455 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:17.706457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:17.706636 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:17.706745 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:17.706750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.707528 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.707531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:17.708110 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:17.708118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.708647 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.708650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:17.708710 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:17.709183 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:17.709195 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:17.711205 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
W20260620 06:12:17.711426 24377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:17.705941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:50683 (local address 127.14.209.190:38257)
0620 06:12:17.706228 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:17.706231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:17.706260 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:12:17.706292 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:17.706295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:17.706333 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:17.706409 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:17.706859 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.707410 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.708305 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.708530 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.708619 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:17.711328 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:50683: BlockingRecv error: recv got EOF from 127.14.209.129:50683 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
Jun 20 06:12:17 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:17 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:17.880919 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:17.875369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:59327)
0620 06:12:17.875636 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:17.875657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:17.875778 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:17.876076 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:17.876079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:17.876283 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:17.876407 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:17.876413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.877231 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.877236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:17.877798 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:17.877806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.878303 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.878307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:17.878390 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:17.878849 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:17.878859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:17.880797 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
W20260620 06:12:17.881068 24377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:17.875420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:59327 (local address 127.14.209.190:38257)
0620 06:12:17.875589 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:17.875593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:17.875730 (+ 137us) server_negotiation.cc:407] Connection header received
0620 06:12:17.875890 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:17.875898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:17.875963 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:17.876054 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:17.876525 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.877126 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.877920 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:17.878192 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:17.878301 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:17.880948 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:59327: BlockingRecv error: recv got EOF from 127.14.209.130:59327 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 06:12:18 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:18 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:18.054687 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:18.049240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:44187)
0620 06:12:18.049513 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:18.049531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:18.049652 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:18.049917 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:18.049919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:18.050193 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:18.050281 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:18.050287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.051201 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.051203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:18.051881 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:18.051900 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.052324 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.052327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:18.052386 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:18.052835 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:18.052845 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:18.054558 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
W20260620 06:12:18.054826 24377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:18.049374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:44187 (local address 127.14.209.190:38257)
0620 06:12:18.049512 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:18.049517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:18.049605 (+ 88us) server_negotiation.cc:407] Connection header received
0620 06:12:18.049755 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:18.049758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:18.049814 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:18.049898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:18.050391 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.051102 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.052007 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.052225 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.052304 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:18.054685 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:44187: BlockingRecv error: recv got EOF from 127.14.209.131:44187 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 06:12:18 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:18 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:18.718439 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:18.712163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:52101)
0620 06:12:18.712451 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:18.712465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:18.712571 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:18.712877 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:18.712879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:18.713081 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:18.713173 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:18.713178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.713941 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.713944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:18.714502 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:18.714511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.714950 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.714953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:18.715019 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:18.715558 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:18.715569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:18.718200 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
W20260620 06:12:18.718528 24381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:18.712213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:52101 (local address 127.14.209.190:38257)
0620 06:12:18.712482 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:18.712486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:18.712517 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:12:18.712693 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:18.712696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:18.712742 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:18.712830 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:18.713295 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.713828 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.714616 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.714838 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.714915 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:18.718365 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:52101: BlockingRecv error: recv got EOF from 127.14.209.129:52101 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
Jun 20 06:12:18 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:18 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:12:18.888504 24381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:18.881937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:46533 (local address 127.14.209.190:38257)
0620 06:12:18.882078 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:18.882082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:18.882302 (+ 220us) server_negotiation.cc:407] Connection header received
0620 06:12:18.882409 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:18.882412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:18.882487 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:18.882580 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:18.883384 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.883921 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.885037 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.885354 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.885420 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:18.888378 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:46533: BlockingRecv error: recv got EOF from 127.14.209.130:46533 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:12:18.888857 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:18.881839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:46533)
0620 06:12:18.882158 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:18.882173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:18.882299 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:18.882824 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:18.882829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:18.883115 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:18.883246 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:18.883254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.884038 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.884041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:18.884894 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:18.884904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:18.885570 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:18.885573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:18.885651 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:18.886224 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:18.886239 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:18.888732 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
Jun 20 06:12:19 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:19 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:19.060842 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:19.055443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:44161)
0620 06:12:19.055728 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:19.055743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:19.055875 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:19.056162 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:19.056165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:19.056368 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:19.056453 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:19.056458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.057207 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.057211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:19.057769 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:19.057777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.058221 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.058225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:19.058309 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:19.058842 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:19.058856 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:19.060674 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
W20260620 06:12:19.060882 24381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:19.055504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:44161 (local address 127.14.209.190:38257)
0620 06:12:19.055674 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:19.055680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:19.055818 (+ 138us) server_negotiation.cc:407] Connection header received
0620 06:12:19.055990 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:19.055993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:19.056052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:19.056139 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:19.056572 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.057099 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.057885 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.058110 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.058218 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:19.060783 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:44161: BlockingRecv error: recv got EOF from 127.14.209.131:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 06:12:19 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:19 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:19.725083 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:19.719698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:36403)
0620 06:12:19.719953 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:19.719967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:19.720061 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:19.720297 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:19.720299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:19.720466 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:19.720558 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:19.720563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.721265 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.721268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:19.722029 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:19.722038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.722519 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.722523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:19.722617 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:19.723199 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:19.723210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:19.724975 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20260620 06:12:19.725311 24385 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:19.719754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:36403 (local address 127.14.209.190:38257)
0620 06:12:19.719974 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:19.719978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:19.720036 (+ 58us) server_negotiation.cc:407] Connection header received
0620 06:12:19.720143 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:19.720146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:19.720187 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:19.720263 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:19.720659 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.721171 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.722162 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.722394 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.722515 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:19.725182 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:36403: BlockingRecv error: recv got EOF from 127.14.209.129:36403 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
Jun 20 06:12:19 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:19 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:19.896519 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:19.890989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:50569)
0620 06:12:19.891295 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:19.891311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:19.891432 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:19.891730 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:19.891733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:19.891941 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:19.892033 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:19.892039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.892835 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.892840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:19.893398 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:19.893406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.893858 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.893861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:19.893946 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:19.894427 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:19.894439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:19.896341 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":83,"threads_started":1}
W20260620 06:12:19.896590 24385 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:19.891140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:50569 (local address 127.14.209.190:38257)
0620 06:12:19.891284 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:19.891289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:19.891390 (+ 101us) server_negotiation.cc:407] Connection header received
0620 06:12:19.891562 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:19.891565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:19.891618 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:19.891712 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:19.892150 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.892724 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.893520 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:19.893746 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:19.893855 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:19.896480 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:50569: BlockingRecv error: recv got EOF from 127.14.209.130:50569 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 06:12:20 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:20 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:20.067411 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:20.061601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:39353)
0620 06:12:20.061959 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:20.061974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:20.062087 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:20.062491 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:20.062494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:20.062685 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:20.062782 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:20.062788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.063533 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.063536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:20.064110 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:20.064119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.064645 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.064648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:20.064712 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:20.065348 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:20.065361 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:20.067291 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
W20260620 06:12:20.067633 24385 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:20.061709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:39353 (local address 127.14.209.190:38257)
0620 06:12:20.061842 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:20.061845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:20.062098 (+ 253us) server_negotiation.cc:407] Connection header received
0620 06:12:20.062167 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:20.062170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:20.062352 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:20.062427 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:20.062894 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.063442 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.064321 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.064551 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.064633 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:20.067487 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:39353: BlockingRecv error: recv got EOF from 127.14.209.131:39353 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 06:12:20 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:20 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:20.730813 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:20.725755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:39907)
0620 06:12:20.726045 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:20.726058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:20.726154 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:20.726400 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:20.726403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:20.726589 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:20.726699 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:20.726704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.727418 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.727420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:20.727980 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:20.727988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.728402 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.728406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:20.728470 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:20.728971 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:20.728982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:20.730695 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":121,"threads_started":1}
W20260620 06:12:20.730935 24389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:20.725885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:39907 (local address 127.14.209.190:38257)
0620 06:12:20.726096 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:20.726100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:20.726112 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:12:20.726240 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:20.726242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:20.726286 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:20.726362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:20.726795 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.727330 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.728072 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.728296 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.728369 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:20.730813 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:39907: BlockingRecv error: recv got EOF from 127.14.209.129:39907 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":53,"threads_started":1}
Jun 20 06:12:20 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:20 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:20.902426 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:20.897256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:37735)
0620 06:12:20.897507 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:20.897520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:20.897604 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:20.897838 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:20.897840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:20.898014 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:20.898106 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:20.898111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.898824 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.898826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:20.899389 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:20.899396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.899773 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.899776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:20.899835 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:20.900295 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:20.900305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:20.902250 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
W20260620 06:12:20.902457 24389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:20.897390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:37735 (local address 127.14.209.190:38257)
0620 06:12:20.897550 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:20.897553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:20.897566 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:20.897669 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:20.897671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:20.897712 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:20.897777 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:20.898230 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.898738 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.899482 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:20.899688 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:20.899758 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:20.902357 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:37735: BlockingRecv error: recv got EOF from 127.14.209.130:37735 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 06:12:21 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:21 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:21.073465 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:21.068169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:54499)
0620 06:12:21.068426 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:21.068439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:21.068533 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:21.068748 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:21.068750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:21.068925 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:21.069014 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:21.069020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.069710 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.069712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:21.070256 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:21.070263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.070764 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.070768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:21.070861 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:21.071488 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:21.071500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:21.073287 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260620 06:12:21.073521 24389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:21.068221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:54499 (local address 127.14.209.190:38257)
0620 06:12:21.068351 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:21.068356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:21.068513 (+ 157us) server_negotiation.cc:407] Connection header received
0620 06:12:21.068609 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:21.068611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:21.068657 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:21.068717 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:21.069111 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.069621 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.070376 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.070643 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.070757 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:21.073404 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:54499: BlockingRecv error: recv got EOF from 127.14.209.131:54499 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:12:21 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:21 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:21.741477 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:21.731528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:33721)
0620 06:12:21.731805 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:21.731819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:21.731907 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:21.732138 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:21.732140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:21.732312 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:21.732400 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:21.732405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.733136 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.733138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:21.733692 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:21.733699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.734103 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.734105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:21.734163 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:21.739283 (+ 5120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:21.739294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:21.741293 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
W20260620 06:12:21.741545 24393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:21.731569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:33721 (local address 127.14.209.190:38257)
0620 06:12:21.731880 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:21.731884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:21.731897 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:12:21.731990 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:21.731992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:21.732032 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:21.732107 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:21.732505 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.733036 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.733787 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.734013 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.734091 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:21.741424 (+ 7333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:33721: BlockingRecv error: recv got EOF from 127.14.209.129:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":72,"threads_started":1}
Jun 20 06:12:21 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:21 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:21.908541 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:21.903280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:55753)
0620 06:12:21.903554 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:21.903567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:21.903662 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:21.903920 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:21.903922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:21.904123 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:21.904220 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:21.904225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.904936 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.904939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:21.905512 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:21.905521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.905983 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.905986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:21.906070 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:21.906558 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:21.906568 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:21.908415 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
W20260620 06:12:21.908705 24393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:21.903343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:55753 (local address 127.14.209.190:38257)
0620 06:12:21.903481 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:21.903487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:21.903659 (+ 172us) server_negotiation.cc:407] Connection header received
0620 06:12:21.903748 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:21.903750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:21.903797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:21.903859 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:21.904322 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.904839 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.905643 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:21.905884 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:21.905963 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:21.908568 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:55753: BlockingRecv error: recv got EOF from 127.14.209.130:55753 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 06:12:22 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:22 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.131@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:22.079728 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:22.074180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.131:42309)
0620 06:12:22.074486 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:22.074498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:22.074628 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:22.074894 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:22.074897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:22.075133 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:22.075222 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:22.075228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.075978 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.075985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:22.076559 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:22.076567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.077015 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.077018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:22.077131 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:22.077629 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:22.077641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:22.079525 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":121,"threads_started":1}
W20260620 06:12:22.079741 24393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:22.074226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:42309 (local address 127.14.209.190:38257)
0620 06:12:22.074361 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:22.074366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:22.074575 (+ 209us) server_negotiation.cc:407] Connection header received
0620 06:12:22.074740 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:22.074743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:22.074796 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:22.074873 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:22.075333 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.075856 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.076667 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.076900 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.077002 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:22.079645 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.131:42309: BlockingRecv error: recv got EOF from 127.14.209.131:42309 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 06:12:22 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:22 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.129@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:22.747372 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:22.742303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.129:39737)
0620 06:12:22.742608 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:22.742627 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:22.742724 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:22.742967 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:22.742969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:22.743155 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:22.743252 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:22.743258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.744019 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.744021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:22.744580 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:22.744587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.744996 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.744999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:22.745059 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:22.745543 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:22.745554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:22.747262 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":59,"threads_started":1}
W20260620 06:12:22.747586 24397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:22.742491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:39737 (local address 127.14.209.190:38257)
0620 06:12:22.742728 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:22.742732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:22.742749 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:12:22.742791 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:22.742794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:22.742844 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:22.742920 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:22.743350 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.743935 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.744680 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.744896 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.744975 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:22.747434 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.129:39737: BlockingRecv error: recv got EOF from 127.14.209.129:39737 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
Jun 20 06:12:22 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:12:22 dist-test-slave-zt01 krb5kdc[23337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.209.130@KRBTEST.COM for oryx/127.14.209.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:12:22.914490 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:22.909415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38257 (local address 127.14.209.130:46253)
0620 06:12:22.909668 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:22.909682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:22.909769 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:22.910028 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:22.910031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:22.910209 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:22.910304 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:22.910309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.911006 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.911008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:22.911632 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:22.911641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.912103 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.912106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:22.912164 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:22.912639 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:22.912650 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:22.914371 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.209.190:38257: Server oryx/127.14.209.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
W20260620 06:12:22.914624 24397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:22.909542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:46253 (local address 127.14.209.190:38257)
0620 06:12:22.909690 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:22.909693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:22.909731 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:12:22.909849 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:22.909851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:22.909892 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:22.909958 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:22.910403 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.910913 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.911764 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:22.912006 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:22.912093 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:22.914515 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.209.130:46253: BlockingRecv error: recv got EOF from 127.14.209.130:46253 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:12:22.973874 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23920
I20260620 06:12:22.979521 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24070
I20260620 06:12:22.984730 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24205
I20260620 06:12:22.989624 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 23850
2026-06-20T06:12:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17841 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T06:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:23Z Disabled control of system clock
I20260620 06:12:23.018507 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40345
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:12:23.121183 24402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:23.121399 24402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:23.121433 24402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:23.125392 24402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:23.125442 24402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:23.125458 24402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:23.125473 24402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:23.125510 24402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:23.130014 24402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:23.130888 24402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:23.132102 24402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:12:23.140574 24402 server_base.cc:1034] running on GCE node
W20260620 06:12:23.140631 24410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.140600 24407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.140842 24408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.141392 24402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:23.142342 24402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:23.143502 24402 hybrid_clock.cc:648] HybridClock initialized: now 1781935943143482 us; error 34 us; skew 500 ppm
I20260620 06:12:23.145198 24402 webserver.cc:460] Webserver started at http://127.14.209.190:35109/ using document root <none> and password file <none>
I20260620 06:12:23.145663 24402 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:23.145716 24402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:23.145910 24402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:23.147746 24402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "937e0907cc2f4c028167b48bf2dd1063"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "0d55aa47bf44558bfdc36f75c5686e82"
server_key_iv: "65bebf68a32d39acf88442b0bd38536e"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.148221 24402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "937e0907cc2f4c028167b48bf2dd1063"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "0d55aa47bf44558bfdc36f75c5686e82"
server_key_iv: "65bebf68a32d39acf88442b0bd38536e"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.151412 24402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:12:23.153622 24416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.155124 24402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:12:23.155242 24402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "937e0907cc2f4c028167b48bf2dd1063"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "0d55aa47bf44558bfdc36f75c5686e82"
server_key_iv: "65bebf68a32d39acf88442b0bd38536e"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.155346 24402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:23.168534 24402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:23.169075 24402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:23.169243 24402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:23.175454 24402 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40345
I20260620 06:12:23.175475 24468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40345 every 8 connection(s)
I20260620 06:12:23.176283 24402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:12:23.179138 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:23.183346 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24402
I20260620 06:12:23.183472 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:12:23.183776 15174 external_mini_cluster.cc:1428] Setting key 277f806d956e7fa1d7e9455fef4244a8
I20260620 06:12:23.184353 24469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063: Bootstrap starting.
I20260620 06:12:23.186551 24469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:23.187278 24469 log.cc:824] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:23.188879 24469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063: No bootstrap required, opened a new log
I20260620 06:12:23.192147 24469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937e0907cc2f4c028167b48bf2dd1063" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40345 } }
I20260620 06:12:23.192354 24469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:23.192425 24469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937e0907cc2f4c028167b48bf2dd1063, State: Initialized, Role: FOLLOWER
I20260620 06:12:23.192622 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.185096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.14.209.190:40345)
0620 06:12:23.185678 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:23.185691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:23.185726 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:12:23.186489 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:23.186513 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:23.186912 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:23.187282 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:23.187664 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.188620 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.189530 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.189982 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.190140 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.190927 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:23.190932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:23.190943 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:23.191363 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:23.191412 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:23.191431 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:23.191534 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:23.191647 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:12:23.191890 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260620 06:12:23.192868 24469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937e0907cc2f4c028167b48bf2dd1063" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40345 } }
I20260620 06:12:23.192993 24469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:12:23.193070 24469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:12:23.193154 24469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:23.194056 24469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937e0907cc2f4c028167b48bf2dd1063" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40345 } }
I20260620 06:12:23.194377 24469 leader_election.cc:302] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937e0907cc2f4c028167b48bf2dd1063; no voters:
I20260620 06:12:23.194674 24469 leader_election.cc:288] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:12:23.194763 24474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:23.195047 24474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [term 1 LEADER]: Becoming Leader. State: Replica: 937e0907cc2f4c028167b48bf2dd1063, State: Running, Role: LEADER
I20260620 06:12:23.195380 24474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937e0907cc2f4c028167b48bf2dd1063" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40345 } }
I20260620 06:12:23.195832 24469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:12:23.196933 24475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937e0907cc2f4c028167b48bf2dd1063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937e0907cc2f4c028167b48bf2dd1063" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40345 } } }
I20260620 06:12:23.197067 24475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:23.196933 24476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937e0907cc2f4c028167b48bf2dd1063. Latest consensus state: current_term: 1 leader_uuid: "937e0907cc2f4c028167b48bf2dd1063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937e0907cc2f4c028167b48bf2dd1063" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40345 } } }
I20260620 06:12:23.197528 24476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063 [sys.catalog]: This master's current role is: LEADER
W20260620 06:12:23.200069 24489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:12:23.200132 24489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:12:23.200207 24483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:23.200853 24483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:23.205924 24483 catalog_manager.cc:1347] Generated new cluster ID: fefccea711384348a5564d5b74be3806
I20260620 06:12:23.205996 24483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:23.245456 24483 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:12:23.246316 24483 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:23.252825 24483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937e0907cc2f4c028167b48bf2dd1063: Generated new TSK 0
I20260620 06:12:23.253398 24483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:12:23.261456 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40345
--builtin_ntp_servers=127.14.209.148:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 06:12:23.377848 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:23.378057 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:23.378089 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:23.382006 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:23.382054 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:23.382146 24493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:12:23.386597 24493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:23.387553 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:23.388770 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:23.397523 24499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.397532 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.397603 24493 server_base.cc:1034] running on GCE node
W20260620 06:12:23.397532 24501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.398185 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:23.398777 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:23.399936 24493 hybrid_clock.cc:648] HybridClock initialized: now 1781935943399900 us; error 48 us; skew 500 ppm
I20260620 06:12:23.401578 24493 webserver.cc:460] Webserver started at http://127.14.209.129:45089/ using document root <none> and password file <none>
I20260620 06:12:23.402103 24493 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:23.402163 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:23.402350 24493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:23.404245 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "87d39abdb6f649c9b630fde6a3dbb781"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "457a42778586d2a50711d633470c6ece"
server_key_iv: "e1c7a94e6b46608537018aaaa2e8aead"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.404757 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "87d39abdb6f649c9b630fde6a3dbb781"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "457a42778586d2a50711d633470c6ece"
server_key_iv: "e1c7a94e6b46608537018aaaa2e8aead"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.408176 24493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:12:23.410494 24507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.411746 24493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:12:23.411870 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "87d39abdb6f649c9b630fde6a3dbb781"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "457a42778586d2a50711d633470c6ece"
server_key_iv: "e1c7a94e6b46608537018aaaa2e8aead"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.411972 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:23.429814 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:23.430446 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:23.430642 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:23.431201 24493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:23.432082 24493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:23.432152 24493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.432219 24493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:23.432261 24493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.440759 24493 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:33699
I20260620 06:12:23.440798 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:33699 every 8 connection(s)
I20260620 06:12:23.441561 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:12:23.447815 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24493
I20260620 06:12:23.447927 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:12:23.448232 15174 external_mini_cluster.cc:1428] Setting key 6f50685dafacf88f2d3bfc196d2644e4
I20260620 06:12:23.450109 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40345
--builtin_ntp_servers=127.14.209.148:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:12:23.450647 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.443148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:52467 (local address 127.14.209.190:40345)
0620 06:12:23.443279 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:23.443282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:23.444229 (+ 947us) server_negotiation.cc:407] Connection header received
0620 06:12:23.445564 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:23.445567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:23.445626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:23.445723 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:23.446527 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.447274 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.448776 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.449005 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.449110 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.450121 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:23.450124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:23.450126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:23.450137 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:23.450156 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:23.450163 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:23.450267 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:23.450488 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 06:12:23.450535 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:12:23.451707 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.443481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40345 (local address 127.14.209.129:52467)
0620 06:12:23.444081 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:23.444117 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:23.445359 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:23.445877 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:23.445883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:23.445956 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:23.446388 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:23.446398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.447396 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.447400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:23.448650 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:23.448659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.449139 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.449143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:23.449303 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.449824 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:12:23.449826 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:23.449876 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:12:23.449964 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:23.450339 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:23.450350 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:23.450359 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:12:23.450486 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 06:12:23.450760 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260620 06:12:23.452836 24621 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40345
I20260620 06:12:23.453037 24621 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:23.453471 24621 heartbeater.cc:511] Master 127.14.209.190:40345 requested a full tablet report, sending...
I20260620 06:12:23.454810 24433 ts_manager.cc:194] Registered new tserver with Master: 87d39abdb6f649c9b630fde6a3dbb781 (127.14.209.129:33699)
I20260620 06:12:23.455912 24433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.129:52467
W20260620 06:12:23.554579 24624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:23.554841 24624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:23.554904 24624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:23.558883 24624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:23.558945 24624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:23.559026 24624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:12:23.563673 24624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:23.564555 24624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:23.565759 24624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:23.574555 24632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.574605 24630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.574613 24629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.574891 24624 server_base.cc:1034] running on GCE node
I20260620 06:12:23.575287 24624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:23.575884 24624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:23.577060 24624 hybrid_clock.cc:648] HybridClock initialized: now 1781935943577032 us; error 39 us; skew 500 ppm
I20260620 06:12:23.578652 24624 webserver.cc:460] Webserver started at http://127.14.209.130:38339/ using document root <none> and password file <none>
I20260620 06:12:23.579232 24624 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:23.579314 24624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:23.579504 24624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:23.581351 24624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "f77fdf3c54f845dbb4a7151ace152fb0"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "958fe10556db6d1cf132931200f592c7"
server_key_iv: "b08bd30ca33cb9d86bdd233d0dd7df03"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.581866 24624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "f77fdf3c54f845dbb4a7151ace152fb0"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "958fe10556db6d1cf132931200f592c7"
server_key_iv: "b08bd30ca33cb9d86bdd233d0dd7df03"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.585129 24624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 06:12:23.587345 24638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.588459 24624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:12:23.588594 24624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "f77fdf3c54f845dbb4a7151ace152fb0"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "958fe10556db6d1cf132931200f592c7"
server_key_iv: "b08bd30ca33cb9d86bdd233d0dd7df03"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.588698 24624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:23.610532 24624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:23.611128 24624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:23.611302 24624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:23.611820 24624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:23.612675 24624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:23.612744 24624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.612807 24624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:23.612854 24624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.621615 24624 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:39441
I20260620 06:12:23.621639 24751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:39441 every 8 connection(s)
I20260620 06:12:23.622390 24624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:12:23.626048 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24624
I20260620 06:12:23.626204 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:12:23.626528 15174 external_mini_cluster.cc:1428] Setting key bfa5cb2f7cf14736db18b9382adfb8ed
I20260620 06:12:23.628394 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40345
--builtin_ntp_servers=127.14.209.148:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:12:23.631224 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.624000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:53601 (local address 127.14.209.190:40345)
0620 06:12:23.624127 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:23.624131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:23.625039 (+ 908us) server_negotiation.cc:407] Connection header received
0620 06:12:23.626600 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:23.626606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:23.626662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:23.626742 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:23.627670 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.628267 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.629418 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.629637 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.629744 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.630747 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:23.630755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:23.630757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:23.630768 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:23.630785 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:23.630793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:23.630869 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:23.631007 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 06:12:23.631083 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:12:23.631966 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.624321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40345 (local address 127.14.209.130:53601)
0620 06:12:23.624900 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:23.624935 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:23.626353 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:23.626931 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:23.626938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:23.627009 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:23.627540 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:23.627552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.628390 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.628398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:23.629297 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:23.629306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.629772 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.629776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:23.629927 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.630507 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:12:23.630509 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:23.630542 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:12:23.630595 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:23.630871 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:23.630881 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:23.630888 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:12:23.631007 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 06:12:23.631265 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260620 06:12:23.632952 24752 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40345
I20260620 06:12:23.633196 24752 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:23.633785 24752 heartbeater.cc:511] Master 127.14.209.190:40345 requested a full tablet report, sending...
I20260620 06:12:23.634824 24433 ts_manager.cc:194] Registered new tserver with Master: f77fdf3c54f845dbb4a7151ace152fb0 (127.14.209.130:39441)
I20260620 06:12:23.635422 24433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.130:53601
W20260620 06:12:23.738718 24755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:23.738935 24755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:23.738970 24755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:23.742874 24755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:23.742920 24755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:23.742996 24755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:12:23.747586 24755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:23.748492 24755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:23.749727 24755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:23.758955 24763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.758960 24760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.759016 24761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.759272 24755 server_base.cc:1034] running on GCE node
I20260620 06:12:23.759706 24755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:23.760332 24755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:23.761536 24755 hybrid_clock.cc:648] HybridClock initialized: now 1781935943761512 us; error 30 us; skew 500 ppm
I20260620 06:12:23.763293 24755 webserver.cc:460] Webserver started at http://127.14.209.131:33751/ using document root <none> and password file <none>
I20260620 06:12:23.763870 24755 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:23.763952 24755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:23.764156 24755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:23.766000 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "0333954efc314a109600ef5970094c06"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "759866db40023304df9bf36797ce4c79"
server_key_iv: "76943e25f69acc3d1347db3df4c3ce1c"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.766513 24755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "0333954efc314a109600ef5970094c06"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "759866db40023304df9bf36797ce4c79"
server_key_iv: "76943e25f69acc3d1347db3df4c3ce1c"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.769829 24755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:12:23.772325 24769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.773425 24755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:12:23.773600 24755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "0333954efc314a109600ef5970094c06"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "759866db40023304df9bf36797ce4c79"
server_key_iv: "76943e25f69acc3d1347db3df4c3ce1c"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.773738 24755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:23.787667 24755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:23.788229 24755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:23.788400 24755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:23.788901 24755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:23.789714 24755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:23.789779 24755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.789844 24755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:23.789887 24755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:23.799196 24755 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:46655
I20260620 06:12:23.799240 24882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:46655 every 8 connection(s)
I20260620 06:12:23.800014 24755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:12:23.804176 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24755
I20260620 06:12:23.804287 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:12:23.804595 15174 external_mini_cluster.cc:1428] Setting key 5fb24cf16a28192ef5b1d94dbde46653
I20260620 06:12:23.809155 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.801688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:47143 (local address 127.14.209.190:40345)
0620 06:12:23.801827 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:23.801830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:23.802801 (+ 971us) server_negotiation.cc:407] Connection header received
0620 06:12:23.804309 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:23.804312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:23.804371 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:23.804447 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:23.805509 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.806142 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.807409 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.807632 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.807738 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.808698 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:23.808702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:23.808705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:23.808721 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:23.808737 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:23.808745 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:23.808818 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:23.808961 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:12:23.809018 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:12:23.809943 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:23.802056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40345 (local address 127.14.209.131:47143)
0620 06:12:23.802665 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:23.802701 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:23.804016 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:23.804645 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:23.804656 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:23.804751 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:23.805332 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:23.805364 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.806265 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.806269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:23.807291 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:23.807301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:23.807777 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:23.807781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:23.807931 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:23.808462 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:12:23.808463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:23.808500 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:12:23.808558 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:23.808820 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:23.808830 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:23.808838 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:12:23.808960 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 06:12:23.809256 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260620 06:12:23.811084 24883 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40345
I20260620 06:12:23.811336 24883 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:23.811830 24883 heartbeater.cc:511] Master 127.14.209.190:40345 requested a full tablet report, sending...
I20260620 06:12:23.812876 24433 ts_manager.cc:194] Registered new tserver with Master: 0333954efc314a109600ef5970094c06 (127.14.209.131:46655)
I20260620 06:12:23.813522 24433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.131:47143
I20260620 06:12:23.818126 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:12:23.820657 24472 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:23.819937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.14.209.190:40345)
0620 06:12:23.820045 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:23.820048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:23.820218 (+ 170us) server_negotiation.cc:407] Connection header received
0620 06:12:23.820308 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:23.820310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:23.820344 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:23.820403 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:23.820567 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40648: BlockingRecv error: recv got EOF from 127.0.0.1:40648 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:12:23.820946 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24493
I20260620 06:12:23.825923 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24624
I20260620 06:12:23.830577 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24755
I20260620 06:12:23.835691 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24402
2026-06-20T06:12:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (844 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T06:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:23Z Disabled control of system clock
I20260620 06:12:23.863850 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46753
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:12:23.968974 24894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:23.969202 24894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:23.969244 24894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:23.973038 24894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:23.973081 24894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:23.973096 24894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:23.973112 24894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:23.973127 24894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:23.977598 24894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:23.978508 24894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:23.979712 24894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:23.987380 24902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.987545 24894 server_base.cc:1034] running on GCE node
W20260620 06:12:23.987419 24899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:23.987414 24900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:23.988245 24894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:23.989236 24894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:23.990433 24894 hybrid_clock.cc:648] HybridClock initialized: now 1781935943990404 us; error 41 us; skew 500 ppm
I20260620 06:12:23.992303 24894 webserver.cc:460] Webserver started at http://127.14.209.190:40083/ using document root <none> and password file <none>
I20260620 06:12:23.992842 24894 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:23.992928 24894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:23.993147 24894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:23.995244 24894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "129d387f95e943b188e10c9bf6635e4b"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "21262d0af5e9288c18ff07c036e056e9"
server_key_iv: "6074941da47573bc361151ff4edb4054"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.995795 24894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "129d387f95e943b188e10c9bf6635e4b"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "21262d0af5e9288c18ff07c036e056e9"
server_key_iv: "6074941da47573bc361151ff4edb4054"
server_key_version: "encryptionkey@0"
I20260620 06:12:23.999264 24894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:12:24.001498 24908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.002626 24894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:24.002775 24894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "129d387f95e943b188e10c9bf6635e4b"
format_stamp: "Formatted at 2026-06-20 06:12:23 on dist-test-slave-zt01"
server_key: "21262d0af5e9288c18ff07c036e056e9"
server_key_iv: "6074941da47573bc361151ff4edb4054"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.002879 24894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:24.007174 24894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:24.007738 24894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:24.007907 24894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:24.014271 24894 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:46753
I20260620 06:12:24.014281 24960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:46753 every 8 connection(s)
I20260620 06:12:24.015156 24894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:12:24.018066 24961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:24.018615 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24894
I20260620 06:12:24.018770 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:12:24.019073 15174 external_mini_cluster.cc:1428] Setting key 0b0c0720dfc302a632d52dea1cca7cc3
I20260620 06:12:24.024358 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.020379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.14.209.190:46753)
0620 06:12:24.020958 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:24.020973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:24.021007 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:12:24.021651 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:24.021666 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:24.021971 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:24.022231 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:24.023165 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:24.023170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:24.023181 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:24.023192 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:24.023245 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:24.023261 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:24.023387 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:24.023493 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 06:12:24.023725 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":138,"threads_started":1}
I20260620 06:12:24.025156 24961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b: Bootstrap starting.
I20260620 06:12:24.027292 24961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:24.028005 24961 log.cc:824] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:24.029798 24961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b: No bootstrap required, opened a new log
I20260620 06:12:24.033334 24961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129d387f95e943b188e10c9bf6635e4b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46753 } }
I20260620 06:12:24.033571 24961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:24.033658 24961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129d387f95e943b188e10c9bf6635e4b, State: Initialized, Role: FOLLOWER
I20260620 06:12:24.034137 24961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129d387f95e943b188e10c9bf6635e4b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46753 } }
I20260620 06:12:24.034284 24961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:12:24.034368 24961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:12:24.034469 24961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:24.035401 24961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129d387f95e943b188e10c9bf6635e4b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46753 } }
I20260620 06:12:24.035720 24961 leader_election.cc:302] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129d387f95e943b188e10c9bf6635e4b; no voters:
I20260620 06:12:24.036010 24961 leader_election.cc:288] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:12:24.036123 24966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:24.036365 24966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [term 1 LEADER]: Becoming Leader. State: Replica: 129d387f95e943b188e10c9bf6635e4b, State: Running, Role: LEADER
I20260620 06:12:24.036687 24966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129d387f95e943b188e10c9bf6635e4b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46753 } }
I20260620 06:12:24.037222 24961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:12:24.038257 24968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129d387f95e943b188e10c9bf6635e4b. Latest consensus state: current_term: 1 leader_uuid: "129d387f95e943b188e10c9bf6635e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129d387f95e943b188e10c9bf6635e4b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46753 } } }
I20260620 06:12:24.038393 24968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:24.038684 24967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "129d387f95e943b188e10c9bf6635e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129d387f95e943b188e10c9bf6635e4b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46753 } } }
I20260620 06:12:24.038770 24967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:24.039685 24975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:24.042008 24975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:24.046657 24975 catalog_manager.cc:1347] Generated new cluster ID: 8ab155c779374d4eb42897910a129a78
I20260620 06:12:24.046738 24975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:24.068192 24975 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:12:24.068490 24975 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:24.074200 24975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 129d387f95e943b188e10c9bf6635e4b: Generated new TSK 0
I20260620 06:12:24.074805 24975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:12:24.079188 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46753
--builtin_ntp_servers=127.14.209.148:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:12:24.188989 24985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:24.189213 24985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:24.189251 24985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:24.193161 24985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:24.193203 24985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:24.193295 24985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:12:24.197841 24985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:24.198727 24985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:24.199985 24985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:24.208983 24991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:24.208977 24993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:24.208997 24990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:24.209148 24985 server_base.cc:1034] running on GCE node
I20260620 06:12:24.209573 24985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:24.210129 24985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:24.211300 24985 hybrid_clock.cc:648] HybridClock initialized: now 1781935944211291 us; error 27 us; skew 500 ppm
I20260620 06:12:24.212915 24985 webserver.cc:460] Webserver started at http://127.14.209.129:37485/ using document root <none> and password file <none>
I20260620 06:12:24.213441 24985 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:24.213536 24985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:24.213740 24985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:24.215687 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "8585b9468d2e4fb68c608254af073f76"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "d1c67562e1e831a3a87bf6e1e16c1e7e"
server_key_iv: "c6f1eef437789262e0840c2f20536685"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.216210 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "8585b9468d2e4fb68c608254af073f76"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "d1c67562e1e831a3a87bf6e1e16c1e7e"
server_key_iv: "c6f1eef437789262e0840c2f20536685"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.219489 24985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:12:24.221690 24999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.222781 24985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:24.223009 24985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "8585b9468d2e4fb68c608254af073f76"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "d1c67562e1e831a3a87bf6e1e16c1e7e"
server_key_iv: "c6f1eef437789262e0840c2f20536685"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.223173 24985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:24.227476 24985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:24.228034 24985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:24.228209 24985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:24.228705 24985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:24.229523 24985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:24.229588 24985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.229648 24985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:24.229694 24985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.238000 24985 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:36759
I20260620 06:12:24.238030 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:36759 every 8 connection(s)
I20260620 06:12:24.238906 24985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:12:24.244679 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.240692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:42065 (local address 127.14.209.190:46753)
0620 06:12:24.240835 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:24.240841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:24.241885 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 06:12:24.243147 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:24.243150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:24.243184 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:24.243260 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:24.244304 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:24.244307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:24.244308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:24.244318 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:24.244334 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:24.244341 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:24.244415 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:24.244536 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 06:12:24.244580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:12:24.245201 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 24985
I20260620 06:12:24.245316 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:12:24.245575 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.241142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46753 (local address 127.14.209.129:42065)
0620 06:12:24.241744 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:24.241779 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:24.242899 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:24.243403 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:24.243409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:24.243486 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:24.244055 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:12:24.244057 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:24.244093 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:12:24.244155 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:24.244421 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:24.244427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:24.244431 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:12:24.244550 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 06:12:24.244719 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260620 06:12:24.245612 15174 external_mini_cluster.cc:1428] Setting key fbec5f48cbc21b898251dccbcb463454
I20260620 06:12:24.246785 25113 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46753
I20260620 06:12:24.247018 25113 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:24.247406 25113 heartbeater.cc:511] Master 127.14.209.190:46753 requested a full tablet report, sending...
I20260620 06:12:24.247711 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46753
--builtin_ntp_servers=127.14.209.148:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 06:12:24.250197 24923 ts_manager.cc:194] Registered new tserver with Master: 8585b9468d2e4fb68c608254af073f76 (127.14.209.129:36759)
W20260620 06:12:24.352746 25116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:24.352991 25116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:24.353063 25116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:24.356948 25116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:24.357028 25116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:24.357112 25116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:12:24.361617 25116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:24.362555 25116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:24.363822 25116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:24.372745 25124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:24.372792 25121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:24.372854 25116 server_base.cc:1034] running on GCE node
W20260620 06:12:24.372753 25122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:24.373368 25116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:24.373975 25116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:24.375139 25116 hybrid_clock.cc:648] HybridClock initialized: now 1781935944375116 us; error 39 us; skew 500 ppm
I20260620 06:12:24.376837 25116 webserver.cc:460] Webserver started at http://127.14.209.130:33551/ using document root <none> and password file <none>
I20260620 06:12:24.377368 25116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:24.377429 25116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:24.377663 25116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:24.379557 25116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "2831d7043de34266ade8bc334e9bb44d"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "ec0d2fd791f5685afef88fcc3ea98a2f"
server_key_iv: "d0855be6328f242e75a7d2c536202c41"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.380084 25116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "2831d7043de34266ade8bc334e9bb44d"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "ec0d2fd791f5685afef88fcc3ea98a2f"
server_key_iv: "d0855be6328f242e75a7d2c536202c41"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.383384 25116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:12:24.385553 25130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.386641 25116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:24.386766 25116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "2831d7043de34266ade8bc334e9bb44d"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "ec0d2fd791f5685afef88fcc3ea98a2f"
server_key_iv: "d0855be6328f242e75a7d2c536202c41"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.386873 25116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:24.390987 25116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:24.391561 25116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:24.391733 25116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:24.392227 25116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:24.393038 25116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:24.393081 25116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.393143 25116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:24.393182 25116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.401439 25116 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:43357
I20260620 06:12:24.401479 25243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:43357 every 8 connection(s)
I20260620 06:12:24.402202 25116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:12:24.403421 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 25116
I20260620 06:12:24.403550 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:12:24.403841 15174 external_mini_cluster.cc:1428] Setting key c62705fdbbdf4270d4d2a5e61483a005
I20260620 06:12:24.405647 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46753
--builtin_ntp_servers=127.14.209.148:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 06:12:24.408332 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.403830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:46841 (local address 127.14.209.190:46753)
0620 06:12:24.403976 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:24.403982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:24.405360 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 06:12:24.406650 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:24.406656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:24.406702 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:24.406796 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:24.407935 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:24.407938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:24.407939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:24.407950 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:24.407968 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:24.407975 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:24.408058 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:24.408207 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 06:12:24.408241 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:12:24.409336 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.404192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46753 (local address 127.14.209.130:46841)
0620 06:12:24.404833 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:24.404878 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:24.406460 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:24.406963 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:24.406972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:24.407106 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:24.407652 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:12:24.407655 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:24.407704 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:12:24.407789 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:24.408068 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:24.408077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:24.408084 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:12:24.408245 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 06:12:24.408451 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":128,"threads_started":1}
I20260620 06:12:24.410398 25244 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46753
I20260620 06:12:24.410611 25244 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:24.410953 25244 heartbeater.cc:511] Master 127.14.209.190:46753 requested a full tablet report, sending...
I20260620 06:12:24.411898 24923 ts_manager.cc:194] Registered new tserver with Master: 2831d7043de34266ade8bc334e9bb44d (127.14.209.130:43357)
W20260620 06:12:24.509761 25247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:24.510016 25247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:24.510089 25247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:24.513965 25247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:24.514036 25247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:24.514124 25247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:12:24.518611 25247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:24.519486 25247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:24.520670 25247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:12:24.529363 25247 server_base.cc:1034] running on GCE node
W20260620 06:12:24.529274 25252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:24.529323 25253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:24.529287 25255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:24.529824 25247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:24.530350 25247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:24.531497 25247 hybrid_clock.cc:648] HybridClock initialized: now 1781935944531478 us; error 31 us; skew 500 ppm
I20260620 06:12:24.533095 25247 webserver.cc:460] Webserver started at http://127.14.209.131:40323/ using document root <none> and password file <none>
I20260620 06:12:24.533603 25247 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:24.533659 25247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:24.533861 25247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:24.535749 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "de460972785744a6a22157f83fb07b4f"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "dfca4470e1d5c0a5c48ff0a8d848aac8"
server_key_iv: "c66c92eccf444ab6d7897ccbe089958c"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.536247 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "de460972785744a6a22157f83fb07b4f"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "dfca4470e1d5c0a5c48ff0a8d848aac8"
server_key_iv: "c66c92eccf444ab6d7897ccbe089958c"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.539779 25247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:12:24.541950 25261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.543118 25247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:12:24.543231 25247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "de460972785744a6a22157f83fb07b4f"
format_stamp: "Formatted at 2026-06-20 06:12:24 on dist-test-slave-zt01"
server_key: "dfca4470e1d5c0a5c48ff0a8d848aac8"
server_key_iv: "c66c92eccf444ab6d7897ccbe089958c"
server_key_version: "encryptionkey@0"
I20260620 06:12:24.543324 25247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:24.547459 25247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:24.547988 25247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:24.548156 25247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:24.548653 25247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:24.549439 25247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:24.549484 25247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.549544 25247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:24.549584 25247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:24.557983 25247 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:39449
I20260620 06:12:24.558010 25374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:39449 every 8 connection(s)
I20260620 06:12:24.558744 25247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:12:24.561517 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 25247
I20260620 06:12:24.561638 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:12:24.561895 15174 external_mini_cluster.cc:1428] Setting key f5e06e5acbffea8feea5da82f26280e2
I20260620 06:12:24.564517 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.560301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:50167 (local address 127.14.209.190:46753)
0620 06:12:24.560435 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:24.560439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:24.561323 (+ 884us) server_negotiation.cc:407] Connection header received
0620 06:12:24.562836 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:24.562840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:24.562880 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:24.562967 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:24.564116 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:24.564119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:24.564120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:12:24.564130 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:24.564150 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:12:24.564157 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:24.564231 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:24.564394 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 06:12:24.564426 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:12:24.565290 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:24.560612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46753 (local address 127.14.209.131:50167)
0620 06:12:24.561195 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:24.561231 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:24.562524 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:24.563191 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:24.563197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:24.563273 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:24.563853 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:12:24.563854 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:24.563910 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:12:24.563987 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:24.564254 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:24.564262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:24.564266 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:12:24.564418 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 06:12:24.564628 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260620 06:12:24.566329 25375 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46753
I20260620 06:12:24.566565 25375 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:24.566929 25375 heartbeater.cc:511] Master 127.14.209.190:46753 requested a full tablet report, sending...
I20260620 06:12:24.567950 24923 ts_manager.cc:194] Registered new tserver with Master: de460972785744a6a22157f83fb07b4f (127.14.209.131:39449)
I20260620 06:12:24.575425 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:12:24.577877 24964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:12:24.577190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.14.209.190:46753)
0620 06:12:24.577319 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:24.577328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:24.577406 (+ 78us) server_negotiation.cc:407] Connection header received
0620 06:12:24.577492 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:24.577495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:24.577520 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:24.577591 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:24.577752 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46612: BlockingRecv error: recv got EOF from 127.0.0.1:46612 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:12:24.577994 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24985
I20260620 06:12:24.583343 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 25116
I20260620 06:12:24.588335 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 25247
I20260620 06:12:24.593405 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 24894
2026-06-20T06:12:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (757 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:12:24 dist-test-slave-zt01 krb5kdc[25385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:12:24 dist-test-slave-zt01 krb5kdc[25385](info): set up 2 sockets
Jun 20 06:12:24 dist-test-slave-zt01 krb5kdc[25385](info): commencing operation
krb5kdc: starting...
W20260620 06:12:25.640010 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:12:25 dist-test-slave-zt01 krb5kdc[25385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:12:25.751853 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46725
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:25.856573 25398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:25.856794 25398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:25.856858 25398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:25.861091 25398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:12:25.861153 25398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:25.861195 25398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:25.861270 25398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:12:25.861318 25398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:12:25.866947 25398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:25.867902 25398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:25.868949 25398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:25.876971 25403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:25.876960 25404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:25.876963 25406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:25.877151 25398 server_base.cc:1034] running on GCE node
I20260620 06:12:25.877732 25398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:25.878664 25398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:25.879817 25398 hybrid_clock.cc:648] HybridClock initialized: now 1781935945879800 us; error 30 us; skew 500 ppm
Jun 20 06:12:25 dist-test-slave-zt01 krb5kdc[25385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935945, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:25.886830 25398 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:12:25.887679 25398 webserver.cc:460] Webserver started at http://127.14.209.190:44361/ using document root <none> and password file <none>
I20260620 06:12:25.888190 25398 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:25.888273 25398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:25.888478 25398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:25.890324 25398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "a09170b78e764dca8f680901ad268f2d"
format_stamp: "Formatted at 2026-06-20 06:12:25 on dist-test-slave-zt01"
server_key: "ccd6e73fda526cbced456768f6ca4ce4"
server_key_iv: "4fe4ead7be71f92a46361bb76ac4fb93"
server_key_version: "encryptionkey@0"
I20260620 06:12:25.890859 25398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "a09170b78e764dca8f680901ad268f2d"
format_stamp: "Formatted at 2026-06-20 06:12:25 on dist-test-slave-zt01"
server_key: "ccd6e73fda526cbced456768f6ca4ce4"
server_key_iv: "4fe4ead7be71f92a46361bb76ac4fb93"
server_key_version: "encryptionkey@0"
I20260620 06:12:25.894207 25398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 06:12:25.896418 25413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:25.897449 25398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:12:25.897578 25398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "a09170b78e764dca8f680901ad268f2d"
format_stamp: "Formatted at 2026-06-20 06:12:25 on dist-test-slave-zt01"
server_key: "ccd6e73fda526cbced456768f6ca4ce4"
server_key_iv: "4fe4ead7be71f92a46361bb76ac4fb93"
server_key_version: "encryptionkey@0"
I20260620 06:12:25.897702 25398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:25.918083 25398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:25.921713 25398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:25.921922 25398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:25.928843 25398 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:46725
I20260620 06:12:25.929150 25465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:46725 every 8 connection(s)
I20260620 06:12:25.929757 25398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:12:25.932430 25466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:25.937018 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 25398
I20260620 06:12:25.937144 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:12:25.937433 15174 external_mini_cluster.cc:1428] Setting key e6fccd15f0784696c76f4d42dce066ce
I20260620 06:12:25.937567 25466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d: Bootstrap starting.
I20260620 06:12:25.939931 25466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:25.940730 25466 log.cc:824] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:25.942564 25466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d: No bootstrap required, opened a new log
Jun 20 06:12:25 dist-test-slave-zt01 krb5kdc[25385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:12:25.946594 25466 raft_consensus.cc:348] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a09170b78e764dca8f680901ad268f2d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46725 } }
I20260620 06:12:25.946805 25466 raft_consensus.cc:374] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:25.946875 25466 raft_consensus.cc:729] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09170b78e764dca8f680901ad268f2d, State: Initialized, Role: FOLLOWER
I20260620 06:12:25.947314 25466 consensus_queue.cc:260] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a09170b78e764dca8f680901ad268f2d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46725 } }
I20260620 06:12:25.947450 25466 raft_consensus.cc:388] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:12:25.947530 25466 raft_consensus.cc:482] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:12:25.947621 25466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:25.948395 25466 raft_consensus.cc:504] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a09170b78e764dca8f680901ad268f2d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46725 } }
I20260620 06:12:25.948704 25466 leader_election.cc:302] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a09170b78e764dca8f680901ad268f2d; no voters:
I20260620 06:12:25.948992 25466 leader_election.cc:288] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:12:25.949055 25471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:25.949236 25471 raft_consensus.cc:686] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [term 1 LEADER]: Becoming Leader. State: Replica: a09170b78e764dca8f680901ad268f2d, State: Running, Role: LEADER
I20260620 06:12:25.949564 25471 consensus_queue.cc:237] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a09170b78e764dca8f680901ad268f2d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46725 } }
I20260620 06:12:25.950143 25466 sys_catalog.cc:564] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:12:25.950657 25473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a09170b78e764dca8f680901ad268f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a09170b78e764dca8f680901ad268f2d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46725 } } }
I20260620 06:12:25.950758 25473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:25.952080 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:25.938712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.14.209.190:46725)
0620 06:12:25.939092 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:25.939140 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:25.939175 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:12:25.939772 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:25.939788 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:25.940132 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:25.940394 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:25.941238 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:25.942288 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:25.943414 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:25.943852 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:25.943969 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:25.946727 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:25.946732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:25.946744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:25.946768 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:25.948613 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:25.949136 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:25.949143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:25.949149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:25.949289 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:25.949600 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:25.949603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:25.949605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:25.950023 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:25.950229 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:25.950776 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 06:12:25.951102 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":124,"threads_started":1}
I20260620 06:12:25.951426 25472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a09170b78e764dca8f680901ad268f2d. Latest consensus state: current_term: 1 leader_uuid: "a09170b78e764dca8f680901ad268f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a09170b78e764dca8f680901ad268f2d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46725 } } }
I20260620 06:12:25.952368 25472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d [sys.catalog]: This master's current role is: LEADER
I20260620 06:12:25.952581 25476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:12:25.955255 25476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:12:25.960542 25476 catalog_manager.cc:1347] Generated new cluster ID: 9f58164367874c9daf933aa39244159a
I20260620 06:12:25.960601 25476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:12:25.978528 25476 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:12:25.979650 25476 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:12:25.992707 25476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d: Generated new TSK 0
I20260620 06:12:25.993364 25476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:12:26.022217 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46725
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:26.131115 25492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:26.131342 25492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:26.131384 25492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:26.135190 25492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:26.135232 25492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:26.135286 25492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:12:26.140219 25492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:26.141016 25492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:26.141891 25492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:26.150725 25498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:26.150725 25497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:26.150839 25492 server_base.cc:1034] running on GCE node
W20260620 06:12:26.150749 25500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:26.151432 25492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:26.152026 25492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:26.153200 25492 hybrid_clock.cc:648] HybridClock initialized: now 1781935946153183 us; error 31 us; skew 500 ppm
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:26.155761 25492 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:12:26.156548 25492 webserver.cc:460] Webserver started at http://127.14.209.129:46867/ using document root <none> and password file <none>
I20260620 06:12:26.157065 25492 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:26.157146 25492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:26.157347 25492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:26.159240 25492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "40cc5cd6cb3b4d4cb65c72f98f978651"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "587e4bdef0bca5ef182361de1451ae78"
server_key_iv: "e43f7e1494ca4c43d8d27ea9246c1658"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.159761 25492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "40cc5cd6cb3b4d4cb65c72f98f978651"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "587e4bdef0bca5ef182361de1451ae78"
server_key_iv: "e43f7e1494ca4c43d8d27ea9246c1658"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.163020 25492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:12:26.165203 25507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.166217 25492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:26.166339 25492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "40cc5cd6cb3b4d4cb65c72f98f978651"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "587e4bdef0bca5ef182361de1451ae78"
server_key_iv: "e43f7e1494ca4c43d8d27ea9246c1658"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.166438 25492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:26.190514 25492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:26.194571 25492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:26.194721 25492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:26.195322 25492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:26.196336 25492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:26.196398 25492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.196463 25492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:26.196492 25492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.208858 25492 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:39129
I20260620 06:12:26.208877 25620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:39129 every 8 connection(s)
I20260620 06:12:26.210126 25492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:12:26.217531 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 25492
I20260620 06:12:26.217659 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:12:26.217947 15174 external_mini_cluster.cc:1428] Setting key 725461f4da968fc532094bf43e7b8452
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:12:26.225059 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.212067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:52545 (local address 127.14.209.190:46725)
0620 06:12:26.212186 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.212203 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.213300 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 06:12:26.214665 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.214669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.214725 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.214802 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:26.215939 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.216468 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.217791 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.218235 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.218349 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.221129 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:26.221133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:26.221135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:26.221162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:26.222776 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.223349 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.223356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.223361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.223458 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.223908 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.223911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.223912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.224085 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:26.224163 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.224685 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.224824 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:12:26.225579 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.212588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46725 (local address 127.14.209.129:52545)
0620 06:12:26.213154 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.213213 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.214473 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.214967 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.214973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.215371 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:26.215803 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.215815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.216596 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.216599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.217582 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.217594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.218498 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.218501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.218694 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.219320 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:26.219341 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:26.220977 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:26.222921 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:26.222925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:26.222934 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:26.223254 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:26.223589 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:26.223595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:26.223600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:26.223798 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:26.224190 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:26.224199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:26.224441 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.224580 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.224806 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":154,"threads_started":1}
I20260620 06:12:26.226656 25621 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46725
I20260620 06:12:26.226881 25621 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:26.227434 25621 heartbeater.cc:511] Master 127.14.209.190:46725 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:12:26.228821 25430 ts_manager.cc:194] Registered new tserver with Master: 40cc5cd6cb3b4d4cb65c72f98f978651 (127.14.209.129:39129)
I20260620 06:12:26.229986 25430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:52545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:12:26.244380 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46725
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:26.346407 25626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:26.346624 25626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:26.346663 25626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:26.350673 25626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:26.350735 25626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:26.350834 25626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:12:26.356132 25626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:26.356938 25626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:26.357760 25626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:26.366351 25631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:26.366350 25634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:26.366591 25626 server_base.cc:1034] running on GCE node
W20260620 06:12:26.366412 25632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:26.367120 25626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:26.367698 25626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:26.368862 25626 hybrid_clock.cc:648] HybridClock initialized: now 1781935946368835 us; error 28 us; skew 500 ppm
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:26.371579 25626 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:12:26.372303 25626 webserver.cc:460] Webserver started at http://127.14.209.130:39361/ using document root <none> and password file <none>
I20260620 06:12:26.372756 25626 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:26.372818 25626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:26.372960 25626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:26.374737 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "ca056ba7055c46e093608a853d29a715"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "cfce63cf0a4359f44e54f187bd081faf"
server_key_iv: "a33f45e91f4c3382cec8e33a5c6028ec"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.375273 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "ca056ba7055c46e093608a853d29a715"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "cfce63cf0a4359f44e54f187bd081faf"
server_key_iv: "a33f45e91f4c3382cec8e33a5c6028ec"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.378522 25626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:12:26.380784 25641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.381899 25626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:12:26.382043 25626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "ca056ba7055c46e093608a853d29a715"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "cfce63cf0a4359f44e54f187bd081faf"
server_key_iv: "a33f45e91f4c3382cec8e33a5c6028ec"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.382148 25626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:26.403231 25626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:26.405678 25626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:26.405846 25626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:26.406332 25626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:26.407037 25626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:26.407145 25626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.407207 25626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:26.407249 25626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.415676 25626 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:34515
I20260620 06:12:26.415706 25754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:34515 every 8 connection(s)
I20260620 06:12:26.416417 25626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:12:26.419296 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 25626
I20260620 06:12:26.419411 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:12:26.419691 15174 external_mini_cluster.cc:1428] Setting key e5e449e5206973de647edbad97223585
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:12:26.430826 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.418162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:54743 (local address 127.14.209.190:46725)
0620 06:12:26.418294 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.418299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.419011 (+ 712us) server_negotiation.cc:407] Connection header received
0620 06:12:26.420505 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.420508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.420561 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.420646 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:26.421727 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.422560 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.424201 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.424628 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.424744 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.427326 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:26.427329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:26.427331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:26.427354 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:26.428706 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.429254 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.429256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.429258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.429351 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.429691 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.429694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.429696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.429862 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:26.429965 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.430502 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.430629 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:12:26.431620 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.418480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46725 (local address 127.14.209.130:54743)
0620 06:12:26.418879 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.418915 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.420336 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.420780 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.420786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.421170 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:26.421589 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.421601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.422707 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.422713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.424059 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.424075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.424763 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.424766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.424897 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.425404 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:26.425419 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:26.427170 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:26.428822 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:26.428830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:26.428844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:26.429158 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:26.429448 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:26.429451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:26.429453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:26.429591 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:26.429961 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:26.429967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:26.430278 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.430447 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.430729 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260620 06:12:26.432617 25755 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46725
I20260620 06:12:26.432832 25755 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:26.433243 25755 heartbeater.cc:511] Master 127.14.209.190:46725 requested a full tablet report, sending...
I20260620 06:12:26.434173 25430 ts_manager.cc:194] Registered new tserver with Master: ca056ba7055c46e093608a853d29a715 (127.14.209.130:34515)
I20260620 06:12:26.435032 25430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:54743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:12:26.444249 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46725
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:12:26.547098 25760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:12:26.547339 25760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:12:26.547415 25760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:12:26.551678 25760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:12:26.551748 25760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:12:26.551838 25760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:12:26.556697 25760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:12:26.557576 25760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:12:26.558431 25760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:12:26.566936 25768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:26.566960 25766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:12:26.567037 25765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:12:26.567212 25760 server_base.cc:1034] running on GCE node
I20260620 06:12:26.567548 25760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:12:26.568112 25760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:12:26.569283 25760 hybrid_clock.cc:648] HybridClock initialized: now 1781935946569262 us; error 29 us; skew 500 ppm
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:12:26.572007 25760 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:12:26.572750 25760 webserver.cc:460] Webserver started at http://127.14.209.131:42435/ using document root <none> and password file <none>
I20260620 06:12:26.573199 25760 fs_manager.cc:359] Metadata directory not provided
I20260620 06:12:26.573249 25760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:12:26.573390 25760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:12:26.575167 25760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "82c1815050af49d78edef3b5f79e1801"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "d9e4d08b7add91c2fa0b3515093801b4"
server_key_iv: "8a7fe8a9d53758edb1972e9608d49d1e"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.575627 25760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "82c1815050af49d78edef3b5f79e1801"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "d9e4d08b7add91c2fa0b3515093801b4"
server_key_iv: "8a7fe8a9d53758edb1972e9608d49d1e"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.578882 25760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:12:26.580999 25775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.582065 25760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:12:26.582196 25760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "82c1815050af49d78edef3b5f79e1801"
format_stamp: "Formatted at 2026-06-20 06:12:26 on dist-test-slave-zt01"
server_key: "d9e4d08b7add91c2fa0b3515093801b4"
server_key_iv: "8a7fe8a9d53758edb1972e9608d49d1e"
server_key_version: "encryptionkey@0"
I20260620 06:12:26.582295 25760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:12:26.593794 25760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:12:26.596623 25760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:12:26.596791 25760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:12:26.597275 25760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:12:26.597968 25760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:12:26.598057 25760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.598120 25760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:12:26.598169 25760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:12:26.606801 25760 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:41375
I20260620 06:12:26.606827 25888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:41375 every 8 connection(s)
I20260620 06:12:26.607583 25760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:12:26.609041 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 25760
I20260620 06:12:26.609179 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:12:26.609473 15174 external_mini_cluster.cc:1428] Setting key f3cefaa150f7bbe8d0211f3f23122b9e
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935946, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:12:26.619925 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.609151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:42053 (local address 127.14.209.190:46725)
0620 06:12:26.609278 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.609284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.610217 (+ 933us) server_negotiation.cc:407] Connection header received
0620 06:12:26.611785 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.611788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.611841 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.611914 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:26.612922 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.613464 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.614530 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.614754 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.614854 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.617035 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:26.617038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:26.617040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:26.617062 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:26.618293 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.618715 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.618718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.618719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.618789 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.619043 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.619045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.619046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.619274 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:26.619365 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.619731 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.619816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:12:26.620570 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.609495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46725 (local address 127.14.209.131:42053)
0620 06:12:26.610091 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.610138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.611604 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.612060 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.612066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.612375 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:12:26.612795 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.612807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.613579 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.613582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.614410 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.614418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.614880 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.614884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.615046 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.615537 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:12:26.615551 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:12:26.616872 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:12:26.618401 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:26.618405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:26.618413 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:26.618635 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:26.618860 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:12:26.618862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:12:26.618863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:12:26.618972 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:12:26.619363 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:12:26.619368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:12:26.619551 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.619688 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.619885 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":108,"threads_started":1}
I20260620 06:12:26.621529 25889 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46725
I20260620 06:12:26.621726 25889 heartbeater.cc:464] Registering TS with master...
I20260620 06:12:26.622225 25889 heartbeater.cc:511] Master 127.14.209.190:46725 requested a full tablet report, sending...
I20260620 06:12:26.623281 25430 ts_manager.cc:194] Registered new tserver with Master: 82c1815050af49d78edef3b5f79e1801 (127.14.209.131:41375)
I20260620 06:12:26.623776 25430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:42053
I20260620 06:12:26.634281 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:12:26.642827 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.636061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.14.209.190:46725)
0620 06:12:26.636191 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.636195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.636330 (+ 135us) server_negotiation.cc:407] Connection header received
0620 06:12:26.636461 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.636465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.636513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.636591 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:12:26.637196 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.637661 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.638432 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.638640 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.638739 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.640066 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:12:26.640069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:12:26.640072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:12:26.640097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:12:26.641254 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.641664 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.641668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.641670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.641742 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:12:26.642007 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:12:26.642010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:12:26.642012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:12:26.642196 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:12:26.642331 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.642571 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.642679 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:12:26.645323 25430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:12:26.648170 25430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:12:26.661540 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.656103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39129 (local address 127.0.0.1:39482)
0620 06:12:26.656826 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.656840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.656948 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.657905 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.657908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.657923 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:26.658041 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.658046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.658989 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.658996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.660214 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.660223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.661040 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.661044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.661155 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.661191 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.661298 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.661396 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":39,"spinlock_wait_cycles":896,"thread_start_us":75,"threads_started":1}
I20260620 06:12:26.661581 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.656973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.14.209.129:39129)
0620 06:12:26.657351 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.657362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.657380 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:12:26.657442 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.657446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.657603 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.657755 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:26.658167 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.658868 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.660334 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.660923 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.661047 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.661165 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.661336 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.661422 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":153,"threads_started":1}
I20260620 06:12:26.662303 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.655959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.14.209.131:41375)
0620 06:12:26.657417 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.657426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.657442 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:12:26.657491 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.657494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.657618 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.657760 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:26.658013 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.659214 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.660884 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.661682 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.661830 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.661942 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.662055 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.662148 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1365,"thread_start_us":122,"threads_started":1}
I20260620 06:12:26.662554 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.656688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.14.209.130:34515)
0620 06:12:26.657227 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.657239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.657264 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:12:26.657412 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.657418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.657611 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.657755 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:26.658179 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.659297 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.660862 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.661665 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.661780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.661892 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.662293 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.662398 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":145,"threads_started":1}
I20260620 06:12:26.662752 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.656509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:34515 (local address 127.0.0.1:50870)
0620 06:12:26.657178 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.657192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.657277 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.657911 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.657914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.657932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:26.658044 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.658055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.659428 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.659431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.660737 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.660748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.661944 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.661947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.662043 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.662073 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.662134 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.662189 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":64,"thread_start_us":97,"threads_started":1}
I20260620 06:12:26.662231 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.655870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:41375 (local address 127.0.0.1:44668)
0620 06:12:26.656356 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.656399 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.656543 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.657758 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.657763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.657783 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:26.657899 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.657905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.659369 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.659375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.660737 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.660748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.661827 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.661830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.661924 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.661944 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.662026 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.662095 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":95,"thread_start_us":68,"threads_started":1}
I20260620 06:12:26.664713 25555 tablet_service.cc:1467] Processing CreateTablet for tablet b9b6a3963b4b478ebfeb44217d10e181 (DEFAULT_TABLE table=test-table [id=1b966466a3e0403dbc2eacd4b06cba78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:26.665146 25689 tablet_service.cc:1467] Processing CreateTablet for tablet b9b6a3963b4b478ebfeb44217d10e181 (DEFAULT_TABLE table=test-table [id=1b966466a3e0403dbc2eacd4b06cba78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:26.665351 25823 tablet_service.cc:1467] Processing CreateTablet for tablet b9b6a3963b4b478ebfeb44217d10e181 (DEFAULT_TABLE table=test-table [id=1b966466a3e0403dbc2eacd4b06cba78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:12:26.665776 25555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b6a3963b4b478ebfeb44217d10e181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:26.666123 25689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b6a3963b4b478ebfeb44217d10e181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:26.666252 25823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b6a3963b4b478ebfeb44217d10e181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:12:26.672741 25906 tablet_bootstrap.cc:492] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: Bootstrap starting.
I20260620 06:12:26.673563 25907 tablet_bootstrap.cc:492] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Bootstrap starting.
I20260620 06:12:26.675239 25907 tablet_bootstrap.cc:654] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:26.675369 25908 tablet_bootstrap.cc:492] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: Bootstrap starting.
I20260620 06:12:26.675494 25906 tablet_bootstrap.cc:654] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:26.676167 25906 log.cc:824] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:26.676440 25907 log.cc:824] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:26.677414 25908 tablet_bootstrap.cc:654] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: Neither blocks nor log segments found. Creating new log.
I20260620 06:12:26.678404 25907 tablet_bootstrap.cc:492] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: No bootstrap required, opened a new log
I20260620 06:12:26.678399 25906 tablet_bootstrap.cc:492] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: No bootstrap required, opened a new log
I20260620 06:12:26.678608 25906 ts_tablet_manager.cc:1397] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:12:26.678617 25907 ts_tablet_manager.cc:1397] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 06:12:26.678658 25908 log.cc:824] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: Log is configured to *not* fsync() on all Append() calls
I20260620 06:12:26.680450 25908 tablet_bootstrap.cc:492] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: No bootstrap required, opened a new log
I20260620 06:12:26.680625 25908 ts_tablet_manager.cc:1397] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:12:26.681515 25907 raft_consensus.cc:348] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.681732 25907 raft_consensus.cc:374] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:26.681806 25907 raft_consensus.cc:729] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c1815050af49d78edef3b5f79e1801, State: Initialized, Role: FOLLOWER
I20260620 06:12:26.682173 25907 consensus_queue.cc:260] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.682288 25906 raft_consensus.cc:348] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.682564 25906 raft_consensus.cc:374] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:26.682657 25906 raft_consensus.cc:729] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca056ba7055c46e093608a853d29a715, State: Initialized, Role: FOLLOWER
I20260620 06:12:26.682739 25889 heartbeater.cc:503] Master 127.14.209.190:46725 was elected leader, sending a full tablet report...
I20260620 06:12:26.683142 25906 consensus_queue.cc:260] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.683796 25907 ts_tablet_manager.cc:1428] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 06:12:26.684212 25908 raft_consensus.cc:348] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.684432 25908 raft_consensus.cc:374] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:12:26.684485 25908 raft_consensus.cc:729] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cc5cd6cb3b4d4cb65c72f98f978651, State: Initialized, Role: FOLLOWER
I20260620 06:12:26.684890 25908 consensus_queue.cc:260] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.685515 25621 heartbeater.cc:503] Master 127.14.209.190:46725 was elected leader, sending a full tablet report...
I20260620 06:12:26.685758 25908 ts_tablet_manager.cc:1428] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:12:26.685990 25906 ts_tablet_manager.cc:1428] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 06:12:26.686240 25755 heartbeater.cc:503] Master 127.14.209.190:46725 was elected leader, sending a full tablet report...
W20260620 06:12:26.711278 25622 tablet.cc:2369] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:12:26.741096 25914 raft_consensus.cc:482] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:12:26.741291 25914 raft_consensus.cc:504] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.742276 25914 leader_election.cc:288] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c1815050af49d78edef3b5f79e1801 (127.14.209.131:41375), ca056ba7055c46e093608a853d29a715 (127.14.209.130:34515)
I20260620 06:12:26.746013 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.742664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:60859 (local address 127.14.209.131:41375)
0620 06:12:26.742778 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.742783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.742879 (+ 96us) server_negotiation.cc:407] Connection header received
0620 06:12:26.743025 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.743028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.743125 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.743206 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:26.743455 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.743959 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.745064 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.745468 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.745597 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.745649 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.745832 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.745893 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:12:26.746016 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.742603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:47655 (local address 127.14.209.130:34515)
0620 06:12:26.742743 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.742751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.742995 (+ 244us) server_negotiation.cc:407] Connection header received
0620 06:12:26.743160 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.743163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.743211 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.743290 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:12:26.743540 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.744028 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.745066 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.745451 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.745574 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.745623 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.745838 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.745901 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:12:26.746044 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.742490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:41375 (local address 127.14.209.129:60859)
0620 06:12:26.742777 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.742805 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.742928 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.743218 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.743220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.743236 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:26.743350 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.743355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.744112 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.744116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.744949 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.744957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.745581 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.745585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.745715 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.745736 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.745818 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.745886 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":31,"thread_start_us":60,"threads_started":1}
I20260620 06:12:26.746032 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.742490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:34515 (local address 127.14.209.129:47655)
0620 06:12:26.742897 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.742909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:12:26.743027 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:12:26.743301 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:12:26.743304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:12:26.743318 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:12:26.743436 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.743441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.744133 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.744139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.744950 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:12:26.744958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.745564 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.745568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:12:26.745676 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.745732 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 06:12:26.745810 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:12:26.745879 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":99,"thread_start_us":111,"threads_started":1}
I20260620 06:12:26.746753 25709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b6a3963b4b478ebfeb44217d10e181" candidate_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca056ba7055c46e093608a853d29a715" is_pre_election: true
I20260620 06:12:26.746773 25843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b6a3963b4b478ebfeb44217d10e181" candidate_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c1815050af49d78edef3b5f79e1801" is_pre_election: true
I20260620 06:12:26.747027 25843 raft_consensus.cc:2454] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40cc5cd6cb3b4d4cb65c72f98f978651 in term 0.
I20260620 06:12:26.747007 25709 raft_consensus.cc:2454] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40cc5cd6cb3b4d4cb65c72f98f978651 in term 0.
I20260620 06:12:26.747459 25511 leader_election.cc:302] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40cc5cd6cb3b4d4cb65c72f98f978651, 82c1815050af49d78edef3b5f79e1801; no voters:
I20260620 06:12:26.747685 25914 raft_consensus.cc:2781] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:12:26.747773 25914 raft_consensus.cc:482] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:12:26.747828 25914 raft_consensus.cc:3037] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:26.748970 25914 raft_consensus.cc:504] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.749280 25914 leader_election.cc:288] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [CANDIDATE]: Term 1 election: Requested vote from peers 82c1815050af49d78edef3b5f79e1801 (127.14.209.131:41375), ca056ba7055c46e093608a853d29a715 (127.14.209.130:34515)
I20260620 06:12:26.749635 25709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b6a3963b4b478ebfeb44217d10e181" candidate_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca056ba7055c46e093608a853d29a715"
I20260620 06:12:26.749629 25843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b6a3963b4b478ebfeb44217d10e181" candidate_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c1815050af49d78edef3b5f79e1801"
I20260620 06:12:26.749738 25843 raft_consensus.cc:3037] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:26.749738 25709 raft_consensus.cc:3037] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:12:26.750969 25843 raft_consensus.cc:2454] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cc5cd6cb3b4d4cb65c72f98f978651 in term 1.
I20260620 06:12:26.750969 25709 raft_consensus.cc:2454] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cc5cd6cb3b4d4cb65c72f98f978651 in term 1.
I20260620 06:12:26.751256 25508 leader_election.cc:302] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40cc5cd6cb3b4d4cb65c72f98f978651, ca056ba7055c46e093608a853d29a715; no voters:
I20260620 06:12:26.751425 25914 raft_consensus.cc:2781] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:12:26.751626 25914 raft_consensus.cc:686] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [term 1 LEADER]: Becoming Leader. State: Replica: 40cc5cd6cb3b4d4cb65c72f98f978651, State: Running, Role: LEADER
I20260620 06:12:26.751925 25914 consensus_queue.cc:237] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } }
I20260620 06:12:26.754626 25429 catalog_manager.cc:5495] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40cc5cd6cb3b4d4cb65c72f98f978651 (127.14.209.129). New cstate: current_term: 1 leader_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40cc5cd6cb3b4d4cb65c72f98f978651" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 } health_report { overall_health: UNKNOWN } } }
I20260620 06:12:26.779906 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.776583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.14.209.129:39129)
0620 06:12:26.776716 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:12:26.776721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:12:26.776735 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:12:26.776896 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:12:26.776898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:12:26.776946 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:12:26.777045 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:12:26.777273 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.777768 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.778620 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:12:26.778824 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:12:26.778925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:12:26.779218 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:12:26.779568 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:12:26.779687 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:12:26.779730 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 06:12:26.779778 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:12:26.785291 25843 raft_consensus.cc:1261] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 1 FOLLOWER]: Refusing update from remote peer 40cc5cd6cb3b4d4cb65c72f98f978651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:26.785375 25709 raft_consensus.cc:1261] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 1 FOLLOWER]: Refusing update from remote peer 40cc5cd6cb3b4d4cb65c72f98f978651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:12:26.785984 25917 consensus_queue.cc:1035] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c1815050af49d78edef3b5f79e1801" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:26.786177 25914 consensus_queue.cc:1035] T b9b6a3963b4b478ebfeb44217d10e181 P 40cc5cd6cb3b4d4cb65c72f98f978651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca056ba7055c46e093608a853d29a715" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:12:26.794376 25922 mvcc.cc:205] Tried to move back new op lower bound from 7298809638025900032 to 7298809637898088448. Current Snapshot: MvccSnapshot[applied={T|T < 7298809638025900032}]
I20260620 06:12:26.801524 25429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:26.801712 25429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:12:26.803990 25429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a09170b78e764dca8f680901ad268f2d: Sending DeleteTablet for 3 replicas of tablet b9b6a3963b4b478ebfeb44217d10e181
I20260620 06:12:26.804791 25555 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b6a3963b4b478ebfeb44217d10e181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:26 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:39482
I20260620 06:12:26.804863 25823 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b6a3963b4b478ebfeb44217d10e181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:26 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:44668
I20260620 06:12:26.805311 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 25492
I20260620 06:12:26.805301 25689 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b6a3963b4b478ebfeb44217d10e181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:12:26 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:50870
I20260620 06:12:26.806249 25930 tablet_replica.cc:330] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: stopping tablet replica
I20260620 06:12:26.806583 25930 raft_consensus.cc:2229] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:26.806878 25930 raft_consensus.cc:2258] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:26.808288 25930 ts_tablet_manager.cc:1905] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:26.811240 25931 tablet_replica.cc:330] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: stopping tablet replica
I20260620 06:12:26.811486 25931 raft_consensus.cc:2229] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:12:26.811735 25931 raft_consensus.cc:2258] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:12:26.812016 25930 ts_tablet_manager.cc:1918] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:12:26.812124 25930 log.cc:1196] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/b9b6a3963b4b478ebfeb44217d10e181
I20260620 06:12:26.812500 25930 ts_tablet_manager.cc:1939] T b9b6a3963b4b478ebfeb44217d10e181 P 82c1815050af49d78edef3b5f79e1801: Deleting consensus metadata
I20260620 06:12:26.812929 25931 ts_tablet_manager.cc:1905] T b9b6a3963b4b478ebfeb44217d10e181 P ca056ba7055c46e093608a853d29a715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:12:26.813764 25417 catalog_manager.cc:4841] TS 82c1815050af49d78edef3b5f79e1801 (127.14.209.131:41375): tablet b9b6a3963b4b478ebfeb44217d10e181 (table test-table [id=1b966466a3e0403dbc2eacd4b06cba78]) successfully deleted
W20260620 06:12:26.814355 25414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:39129 (error 108)
I20260620 06:12:26.814553 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 25626
I20260620 06:12:26.816326 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.815246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:39129 (local address 127.0.0.1:39502)
0620 06:12:26.816200 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.816259 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":878}
W20260620 06:12:26.816474 25414 catalog_manager.cc:4584] TS 40cc5cd6cb3b4d4cb65c72f98f978651 (127.14.209.129:39129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9b6a3963b4b478ebfeb44217d10e181: Network error: Client connection negotiation failed: client connection to 127.14.209.129:39129: connect: Connection refused (error 111)
I20260620 06:12:26.821697 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 25760
I20260620 06:12:26.823272 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:12:26.821849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:34515 (local address 127.0.0.1:50880)
0620 06:12:26.823108 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0620 06:12:26.823190 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.130:34515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1148}
I20260620 06:12:26.826982 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 25398
2026-06-20T06:12:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2235 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25935](info): set up 2 sockets
Jun 20 06:12:26 dist-test-slave-zt01 krb5kdc[25935](info): commencing operation
krb5kdc: starting...
W20260620 06:12:27.868299 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:12:27 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:12:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/rangeradmin_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/rangeradmin_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/rangerlookup_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/rangerlookup_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
I20260620 06:12:28.023823 15174 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskepqMJ0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/postgres -l logfile start
2026-06-20 06:12:30.516 UTC [25965] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:12:30.516 UTC [25965] LOG: listening on IPv4 address "127.14.209.148", port 43391
2026-06-20 06:12:30.520 UTC [25965] LOG: listening on Unix socket "/tmp/.s.PGSQL.43391"
2026-06-20 06:12:30.555 UTC [25968] LOG: database system was shut down at 2026-06-20 06:12:29 UTC
2026-06-20 06:12:30.559 UTC [25965] LOG: database system is ready to accept connections
I20260620 06:12:31.503868 15174 mini_postgres.cc:96] Postgres bound to 43391
2026-06-20 06:12:31.511 UTC [25976] FATAL: database "slave" does not exist
127.14.209.148:43391 - accepting connections
I20260620 06:12:31.511775 15174 mini_ranger.cc:161] Starting Ranger...
I20260620 06:12:31.526587 15174 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:12:32.032963 15174 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:12:32.033064 15174 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-admin
2026-06-20 06:12:32,300 [I] DB FLAVOR :POSTGRES
2026-06-20 06:12:32,300 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:12:32,301 [I] Checking connection..
2026-06-20 06:12:32,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:12:32,545 [I] Checking connection passed.
2026-06-20 06:12:32,545 [I] --------- Verifying version history table ---------
2026-06-20 06:12:32,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:12:32,796 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:12:32,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:12:33,030 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:12:33,030 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:12:33,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:12:33.249 UTC [26040] WARNING: there is no transaction in progress
2026-06-20 06:12:33,255 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:12:33,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:12:33,506 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:12:33,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:12:33,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:12:33,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:12:33,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zt01.c.gcp-upstream.internal','N') ;"
2026-06-20 06:12:34,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:12:34,468 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:12:34,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:12:34,706 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:12:34,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:12:34,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:12:35,185 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:12:35,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:12:36.154 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.171 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.188 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.203 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.219 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.240 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.302 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.310 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.328 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.338 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.349 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.363 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.373 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.379 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.386 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:36.392 UTC [26148] WARNING: there is no transaction in progress
2026-06-20 06:12:37,040 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:12:37,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:12:37,275 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:12:37,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:12:37,526 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:12:37,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:12:37,770 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:12:37,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:12:38,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:12:38,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zt01.c.gcp-upstream.internal';"
2026-06-20 06:12:38,498 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:12:38,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:12:38,734 [I] DB_PATCHES have already been applied
I20260620 06:12:38.740104 15174 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-admin:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskepqMJ0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:12:38.740160 15174 mini_ranger.cc:193] Using host: 127.14.209.148
I20260620 06:12:38.743376 15174 mini_ranger.cc:227] Ranger admin URL: http://127.14.209.148:45637
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:12:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:12:39.853442 15174 mini_ranger.cc:160] Time spent starting Ranger: real 8.342s user 0.000s sys 0.010s
Jun 20, 2026 6:12:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:12:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:12:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:12:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:12:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:13:00.238722 15174 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/rangerkms_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/rangerkms_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.209.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.209.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:13:00 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935980, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:13:00.328120 15174 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:13:00.336799 15174 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:13:01.187753 15174 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:13:01.187855 15174 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms
I20260620 06:13:01.187978 15174 mini_ranger_kms.cc:227] Using postgres at 127.14.209.148:43391
2026-06-20 06:13:01,379 [I] DB FLAVOR :POSTGRES
2026-06-20 06:13:01,379 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:13:01,379 [I] Checking connection
2026-06-20 06:13:01,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:13:01,636 [I] connection success
2026-06-20 06:13:01,636 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:13:01,636 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:13:01,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:13:01,898 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:13:01,899 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:13:01,899 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:13:01,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:43391/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:13:02,153 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:13:02.623171 15174 mini_ranger_kms.cc:310] Created kms service
I20260620 06:13:02.819906 15174 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:13:02.864023 15174 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:13:02 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935980, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:03.251986 15174 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:13:03.252135 15174 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms:/tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:13:03.252158 15174 mini_ranger_kms.cc:240] Using host: 127.14.209.148
I20260620 06:13:03.256342 15174 mini_ranger_kms.cc:276] Ranger KMS PID: 26328
I20260620 06:13:03.256460 15174 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.209.148:54703
Jun 20, 2026 6:13:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:13:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:13:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:13:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:13:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:13:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 06:13:05 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.209.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:05.478897 15174 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.151s user 0.000s sys 0.173s
I20260620 06:13:05.479051 15174 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:13:05.479183 15174 mini_ranger_kms.cc:401] 127.14.209.148:54703/kms/v1/keys
Jun 20 06:13:05 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.209.148@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
Jun 20 06:13:06 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.209.148@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
Jun 20 06:13:08 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:13:08.563575 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39093
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:39093
--ranger_config_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:54703/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:08.671350 26379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:08.671573 26379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:08.671610 26379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:08.675513 26379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:13:08.675561 26379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:08.675580 26379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:08.675596 26379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:13:08.675630 26379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:13:08.681231 26379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.209.148:54703/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:39093
--ranger_config_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:08.682142 26379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:08.683141 26379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:13:08.692399 26379 server_base.cc:1034] running on GCE node
W20260620 06:13:08.692392 26384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:08.692380 26385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:08.692817 26387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:08.693337 26379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:08.694290 26379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:08.695464 26379 hybrid_clock.cc:648] HybridClock initialized: now 1781935988695439 us; error 43 us; skew 500 ppm
Jun 20 06:13:08 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935988, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:08.698235 26379 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:13:08.699110 26379 webserver.cc:460] Webserver started at http://127.14.209.190:39341/ using document root <none> and password file <none>
I20260620 06:13:08.699615 26379 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:08.699666 26379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:08.699867 26379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:08 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935988, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:08.753077 26379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "5e28566a69dd4ea9897a5d711370233d"
format_stamp: "Formatted at 2026-06-20 06:13:08 on dist-test-slave-zt01"
server_key: "f808caa694996495392fa0a0c1cb7d05"
server_key_iv: "7a63e521b861e063a78f108bc3fc1159"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:08.753757 26379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "5e28566a69dd4ea9897a5d711370233d"
format_stamp: "Formatted at 2026-06-20 06:13:08 on dist-test-slave-zt01"
server_key: "f808caa694996495392fa0a0c1cb7d05"
server_key_iv: "7a63e521b861e063a78f108bc3fc1159"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:08.758389 26379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:13:08.774427 26394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:08.775735 26379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:13:08.775903 26379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "5e28566a69dd4ea9897a5d711370233d"
format_stamp: "Formatted at 2026-06-20 06:13:08 on dist-test-slave-zt01"
server_key: "f808caa694996495392fa0a0c1cb7d05"
server_key_iv: "7a63e521b861e063a78f108bc3fc1159"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:08.776026 26379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:08.794548 26379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:08.797362 26379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:08.797545 26379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:08.805410 26379 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:39093
I20260620 06:13:08.805460 26446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:39093 every 8 connection(s)
I20260620 06:13:08.806319 26379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:13:08.808974 26447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:08.810071 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 26379
I20260620 06:13:08.810197 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:13:08.810514 15174 external_mini_cluster.cc:1428] Setting key d222e08cbeb34ebf13058a8aebe1572f
I20260620 06:13:08.815332 26447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d: Bootstrap starting.
I20260620 06:13:08.818590 26447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:08.819633 26447 log.cc:824] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:13:08 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:08.821945 26447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d: No bootstrap required, opened a new log
I20260620 06:13:08.826175 26447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e28566a69dd4ea9897a5d711370233d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39093 } }
I20260620 06:13:08.826437 26447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:08.826521 26447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e28566a69dd4ea9897a5d711370233d, State: Initialized, Role: FOLLOWER
I20260620 06:13:08.827014 26447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e28566a69dd4ea9897a5d711370233d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39093 } }
I20260620 06:13:08.827201 26447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:13:08.827292 26447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:13:08.827390 26447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:08.828572 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:08.811936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.14.209.190:39093)
0620 06:13:08.812383 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:08.812401 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:08.812441 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:13:08.813279 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:08.813303 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:08.813684 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:08.814028 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:08.814903 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:08.816439 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:08.817898 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:08.818542 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:08.821716 (+ 3174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:08.821909 (+ 193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:08.821915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:08.821930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:08.821950 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:08.824599 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:08.825412 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:08.825418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:08.825424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:08.825550 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:08.825901 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:08.825904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:08.825906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:08.826357 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:08.826563 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:08.826948 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 06:13:08.827252 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":137,"threads_started":1}
I20260620 06:13:08.828881 26447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e28566a69dd4ea9897a5d711370233d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39093 } }
I20260620 06:13:08.829206 26447 leader_election.cc:302] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e28566a69dd4ea9897a5d711370233d; no voters:
I20260620 06:13:08.829628 26452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:13:08.829854 26452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [term 1 LEADER]: Becoming Leader. State: Replica: 5e28566a69dd4ea9897a5d711370233d, State: Running, Role: LEADER
I20260620 06:13:08.830186 26452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e28566a69dd4ea9897a5d711370233d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39093 } }
I20260620 06:13:08.829507 26447 leader_election.cc:288] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:13:08.831743 26447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:13:08.831982 26453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e28566a69dd4ea9897a5d711370233d. Latest consensus state: current_term: 1 leader_uuid: "5e28566a69dd4ea9897a5d711370233d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e28566a69dd4ea9897a5d711370233d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39093 } } }
I20260620 06:13:08.832083 26453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [sys.catalog]: This master's current role is: LEADER
I20260620 06:13:08.832537 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e28566a69dd4ea9897a5d711370233d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e28566a69dd4ea9897a5d711370233d" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39093 } } }
I20260620 06:13:08.832672 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d [sys.catalog]: This master's current role is: LEADER
I20260620 06:13:08.834332 26447 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:13:08.835359 26456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:13:08.840121 26456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:13:08.846246 26456 catalog_manager.cc:1347] Generated new cluster ID: ad9ced9c17f24caa8ca0c8016eb2957b
I20260620 06:13:08.846294 26456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:13:08.863873 26456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:13:08.865101 26456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:13:08.870332 26456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d: Generated new TSK 0
I20260620 06:13:08.871084 26456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:13:09.596201 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:54703/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:39093
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:09.738351 26492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:09.738598 26492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:09.738720 26492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:09.742830 26492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:09.742905 26492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:09.743036 26492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:13:09.748188 26492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.209.148:54703/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:09.749091 26492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:09.750043 26492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:09.760524 26499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:09.760548 26500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:09.760545 26502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:09.761270 26492 server_base.cc:1034] running on GCE node
I20260620 06:13:09.761710 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:09.762403 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:09.763602 26492 hybrid_clock.cc:648] HybridClock initialized: now 1781935989763566 us; error 60 us; skew 500 ppm
Jun 20 06:13:09 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:09.767031 26492 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:13:09.768030 26492 webserver.cc:460] Webserver started at http://127.14.209.129:43755/ using document root <none> and password file <none>
I20260620 06:13:09.768688 26492 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:09.768776 26492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:09.768987 26492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:09 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:09.789479 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "e340a9f2fdb447e4b747301b280ed359"
format_stamp: "Formatted at 2026-06-20 06:13:09 on dist-test-slave-zt01"
server_key: "97931c0daab9e9e5d90bdb5b04d90cf1"
server_key_iv: "0da5e9f4fe5f48a387f9ac2293c08adc"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:09.790195 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "e340a9f2fdb447e4b747301b280ed359"
format_stamp: "Formatted at 2026-06-20 06:13:09 on dist-test-slave-zt01"
server_key: "97931c0daab9e9e5d90bdb5b04d90cf1"
server_key_iv: "0da5e9f4fe5f48a387f9ac2293c08adc"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:09.796790 26492 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260620 06:13:09.825800 26509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:09.827297 26492 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260620 06:13:09.827416 26492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "e340a9f2fdb447e4b747301b280ed359"
format_stamp: "Formatted at 2026-06-20 06:13:09 on dist-test-slave-zt01"
server_key: "97931c0daab9e9e5d90bdb5b04d90cf1"
server_key_iv: "0da5e9f4fe5f48a387f9ac2293c08adc"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:09.827540 26492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:09.867211 26492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:09.870576 26492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:09.870755 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:09.871443 26492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:09.872401 26492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:09.872473 26492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:09.872540 26492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:09.872571 26492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:09.886049 26492 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:43257
I20260620 06:13:09.887145 26492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:13:09.887645 26624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:43257 every 8 connection(s)
I20260620 06:13:09.893478 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 26492
I20260620 06:13:09.893607 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:13:09.893914 15174 external_mini_cluster.cc:1428] Setting key bdb936278093c3cff321f1712ef326db
Jun 20 06:13:09 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:09.913434 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:09.889738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:33333 (local address 127.14.209.190:39093)
0620 06:13:09.899222 (+ 9484us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:09.899227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:09.899243 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:13:09.899281 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:09.899284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:09.899330 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:09.899424 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:09.900906 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:09.901748 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:09.903406 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:09.903768 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:09.903873 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:09.906823 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:09.906829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:09.906833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:09.906859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:09.908780 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:09.910998 (+ 2218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:09.911001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:09.911003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:09.911114 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:09.911499 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:09.911503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:09.911504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:09.911713 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:09.911794 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:09.913155 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0620 06:13:09.913272 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9408,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:13:09.918939 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:09.890203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:39093 (local address 127.14.209.129:33333)
0620 06:13:09.891483 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:09.891535 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:09.892507 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:09.899730 (+ 7223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:09.899739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:09.900078 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:09.900731 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:09.900750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:09.901896 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:09.901899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:09.903266 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:09.903277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:09.904038 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:09.904042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:09.904229 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:09.904917 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:09.904937 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:09.906619 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:09.910516 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:09.910522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:09.910535 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:09.910876 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:09.911232 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:09.911234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:09.911236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:09.911394 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:09.911854 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:09.911860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:09.912137 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 06:13:09.912330 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 06:13:09.912616 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":127,"threads_started":1}
I20260620 06:13:09.920545 26625 heartbeater.cc:347] Connected to a master server at 127.14.209.190:39093
I20260620 06:13:09.920780 26625 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:09.921314 26625 heartbeater.cc:511] Master 127.14.209.190:39093 requested a full tablet report, sending...
I20260620 06:13:09.922978 26411 ts_manager.cc:194] Registered new tserver with Master: e340a9f2fdb447e4b747301b280ed359 (127.14.209.129:43257)
I20260620 06:13:09.924484 26411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:33333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Jun 20 06:13:09 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:13:09.947933 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:54703/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:39093
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:10.101971 26631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:10.102387 26631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:10.102483 26631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:10.108022 26631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:10.108134 26631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:10.108260 26631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:13:10.115688 26631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.209.148:54703/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:10.117092 26631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:10.118326 26631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:10.130540 26637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:10.133199 26638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:10.138945 26640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:10.139350 26631 server_base.cc:1034] running on GCE node
I20260620 06:13:10.140492 26631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:10.141274 26631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:10.142554 26631 hybrid_clock.cc:648] HybridClock initialized: now 1781935990142531 us; error 36 us; skew 500 ppm
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:10.145750 26631 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:13:10.146661 26631 webserver.cc:460] Webserver started at http://127.14.209.130:33747/ using document root <none> and password file <none>
I20260620 06:13:10.147293 26631 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:10.147380 26631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:10.147588 26631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:10.163014 26631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "2f40d69ee90644899b666c870536d89c"
format_stamp: "Formatted at 2026-06-20 06:13:10 on dist-test-slave-zt01"
server_key: "e73121654ee8c7b0561c3dd0870a7866"
server_key_iv: "171ea1686a6d67d1c02e406ee260266c"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:10.163841 26631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "2f40d69ee90644899b666c870536d89c"
format_stamp: "Formatted at 2026-06-20 06:13:10 on dist-test-slave-zt01"
server_key: "e73121654ee8c7b0561c3dd0870a7866"
server_key_iv: "171ea1686a6d67d1c02e406ee260266c"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:10.168424 26631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:13:10.187439 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:10.188947 26631 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260620 06:13:10.189069 26631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "2f40d69ee90644899b666c870536d89c"
format_stamp: "Formatted at 2026-06-20 06:13:10 on dist-test-slave-zt01"
server_key: "e73121654ee8c7b0561c3dd0870a7866"
server_key_iv: "171ea1686a6d67d1c02e406ee260266c"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:10.189200 26631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:10.207444 26631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:10.210858 26631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:10.211042 26631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:10.211692 26631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:10.212589 26631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:10.212647 26631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:10.212709 26631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:10.212738 26631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:10.225875 26631 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:37135
I20260620 06:13:10.226918 26631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:13:10.228531 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 26631
I20260620 06:13:10.228653 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:13:10.228941 15174 external_mini_cluster.cc:1428] Setting key cd1b0b4f64c2ed9a7c3617faad20524c
I20260620 06:13:10.241241 26760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:37135 every 8 connection(s)
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:13:10.257865 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.244430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:60289 (local address 127.14.209.190:39093)
0620 06:13:10.244550 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.244555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.245258 (+ 703us) server_negotiation.cc:407] Connection header received
0620 06:13:10.246444 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.246448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.246502 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.246554 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:10.248137 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.248995 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.250621 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.250984 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.251086 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.253886 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:10.253890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:10.253893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:10.253918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:10.255601 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:10.256164 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:10.256167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:10.256169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:10.256255 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:10.256616 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:10.256620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:10.256621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:10.256832 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:10.256908 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.257526 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.257657 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:13:10.259306 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.243422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:39093 (local address 127.14.209.130:60289)
0620 06:13:10.245116 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.245166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.246205 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.246935 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.246944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.247327 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:10.247971 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.247988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.249196 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.249199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.250475 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.250486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.251278 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.251281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.251467 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.252143 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:10.252161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:10.253708 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:10.255736 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:10.255742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:10.255756 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:10.256053 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:10.256356 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:10.256359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:10.256361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:10.256514 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:10.256970 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:10.256976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:10.257265 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.258023 (+ 758us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.258329 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"thread_start_us":128,"threads_started":1}
I20260620 06:13:10.260516 26761 heartbeater.cc:347] Connected to a master server at 127.14.209.190:39093
I20260620 06:13:10.260759 26761 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:10.261274 26761 heartbeater.cc:511] Master 127.14.209.190:39093 requested a full tablet report, sending...
I20260620 06:13:10.262251 26411 ts_manager.cc:194] Registered new tserver with Master: 2f40d69ee90644899b666c870536d89c (127.14.209.130:37135)
I20260620 06:13:10.262956 26411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:60289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:13:10.279119 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:54703/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:39093
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:10.449663 26766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:10.449915 26766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:10.449986 26766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:10.455735 26766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:10.455813 26766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:10.455917 26766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:13:10.462142 26766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.209.148:54703/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:10.463299 26766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:10.464264 26766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:10.476763 26774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:10.476780 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:10.477984 26766 server_base.cc:1034] running on GCE node
W20260620 06:13:10.477931 26771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:10.478469 26766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:10.479223 26766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:10.480418 26766 hybrid_clock.cc:648] HybridClock initialized: now 1781935990480349 us; error 80 us; skew 500 ppm
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:10.483855 26766 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:13:10.485183 26766 webserver.cc:460] Webserver started at http://127.14.209.131:38447/ using document root <none> and password file <none>
I20260620 06:13:10.485828 26766 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:10.485901 26766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:10.486109 26766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:10.520223 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "5afc8b0034ab4582b288cb3a4249c284"
format_stamp: "Formatted at 2026-06-20 06:13:10 on dist-test-slave-zt01"
server_key: "263d71f6e33973d923425f946490686b"
server_key_iv: "5b82286a046f1a66e3bfbcd66bd0af9c"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:10.520936 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "5afc8b0034ab4582b288cb3a4249c284"
format_stamp: "Formatted at 2026-06-20 06:13:10 on dist-test-slave-zt01"
server_key: "263d71f6e33973d923425f946490686b"
server_key_iv: "5b82286a046f1a66e3bfbcd66bd0af9c"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:10.527597 26766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 06:13:10.548717 26781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:10.554828 26766 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260620 06:13:10.554963 26766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "5afc8b0034ab4582b288cb3a4249c284"
format_stamp: "Formatted at 2026-06-20 06:13:10 on dist-test-slave-zt01"
server_key: "263d71f6e33973d923425f946490686b"
server_key_iv: "5b82286a046f1a66e3bfbcd66bd0af9c"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:10.555105 26766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:10.583387 26766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:10.586598 26766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:10.586776 26766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:10.587395 26766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:10.588285 26766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:10.588342 26766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:10.588390 26766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:10.588419 26766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:10.600755 26766 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:39655
I20260620 06:13:10.601706 26766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:13:10.603444 26894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:39655 every 8 connection(s)
I20260620 06:13:10.609576 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 26766
I20260620 06:13:10.609715 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:13:10.610034 15174 external_mini_cluster.cc:1428] Setting key 0c175bdcc91359f3096875be4eba4241
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935990, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:10.618387 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.604346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:46865 (local address 127.14.209.190:39093)
0620 06:13:10.604463 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.604468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.605869 (+ 1401us) server_negotiation.cc:407] Connection header received
0620 06:13:10.606973 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.606976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.607029 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.607102 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:10.608550 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.609315 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.610847 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.611189 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.611269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.614393 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:10.614397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:10.614399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:10.614426 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:10.616143 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:10.616728 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:10.616732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:10.616734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:10.616818 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:10.617214 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:10.617217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:10.617219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:10.617423 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:10.617495 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.618110 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.618237 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:13:10.619824 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.604717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:39093 (local address 127.14.209.131:46865)
0620 06:13:10.605728 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.605777 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.606746 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.607475 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.607483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.607783 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:10.608380 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.608399 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.609468 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.609471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.610714 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.610724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.611450 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.611454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.611641 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.612601 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:10.612621 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:10.614209 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:10.616287 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:10.616293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:10.616306 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:10.616613 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:10.616922 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:10.616925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:10.616927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:10.617090 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:10.617568 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:10.617574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:10.617848 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.618586 (+ 738us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.618882 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":126,"threads_started":1}
I20260620 06:13:10.620913 26895 heartbeater.cc:347] Connected to a master server at 127.14.209.190:39093
I20260620 06:13:10.621121 26895 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:10.621606 26895 heartbeater.cc:511] Master 127.14.209.190:39093 requested a full tablet report, sending...
I20260620 06:13:10.622689 26411 ts_manager.cc:194] Registered new tserver with Master: 5afc8b0034ab4582b288cb3a4249c284 (127.14.209.131:39655)
I20260620 06:13:10.623378 26411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:46865
I20260620 06:13:10.625958 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:13:10.638783 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.628643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.14.209.190:39093)
0620 06:13:10.628765 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.628771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.629406 (+ 635us) server_negotiation.cc:407] Connection header received
0620 06:13:10.629975 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.629978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.630021 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.630070 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:10.631111 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.631956 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.633655 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.634021 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.634093 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.635805 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:10.635809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:10.635811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:10.635833 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:10.637216 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:10.637621 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:10.637627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:10.637629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:10.637711 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:10.637979 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:10.637983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:10.637985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:10.638143 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:10.638279 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.638497 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.638598 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:13:10.641511 26411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:13:10.644850 26411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:13:10.678843 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.665704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:39655 (local address 127.0.0.1:48512)
0620 06:13:10.668097 (+ 2393us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.668108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.668186 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.668860 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.668866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.668887 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:10.668995 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.669000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.674519 (+ 5519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.674522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.675795 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.675804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.678346 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.678350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.678450 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.678483 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.678583 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.678696 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343,"thread_start_us":69,"threads_started":1}
I20260620 06:13:10.681718 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.664346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.14.209.129:43257)
0620 06:13:10.666492 (+ 2146us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.666500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.668787 (+ 2287us) server_negotiation.cc:407] Connection header received
0620 06:13:10.669220 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.669226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.669391 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.670341 (+ 950us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:10.670468 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.671625 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.678972 (+ 7347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.679727 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.679839 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.679938 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.681472 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.681570 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2046,"thread_start_us":112,"threads_started":1}
I20260620 06:13:10.681880 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.664718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:37135 (local address 127.0.0.1:43636)
0620 06:13:10.667606 (+ 2888us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.667618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.667736 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.672011 (+ 4275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.672014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.672031 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:10.672130 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.672136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.676772 (+ 4636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.676776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.678011 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.678022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.679269 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.679273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.679390 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.679412 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.679487 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.679549 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2829,"thread_start_us":82,"threads_started":1}
I20260620 06:13:10.682214 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.663376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:43257 (local address 127.0.0.1:38484)
0620 06:13:10.666722 (+ 3346us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.666765 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.669094 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.670131 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.670134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.670149 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:10.670231 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.670236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.671758 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.671761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.677118 (+ 5357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.677140 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.680069 (+ 2929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.680073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.680162 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.680183 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.680245 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.680302 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3182,"thread_start_us":82,"threads_started":1}
I20260620 06:13:10.682629 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.664876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.14.209.130:37135)
0620 06:13:10.671466 (+ 6590us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.671477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.671499 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:13:10.671567 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.671572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.671732 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.671875 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:10.675483 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.676631 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.678181 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.678961 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.679097 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.679208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.682394 (+ 3186us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.682494 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6488,"thread_start_us":112,"threads_started":1}
I20260620 06:13:10.685248 26559 tablet_service.cc:1467] Processing CreateTablet for tablet 2526aff06f5c4e70ba0ad488789723c8 (DEFAULT_TABLE table=test-table [id=474d5f03aba84dbaae5f5c96527310cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:13:10.688141 26691 tablet_service.cc:1467] Processing CreateTablet for tablet 2526aff06f5c4e70ba0ad488789723c8 (DEFAULT_TABLE table=test-table [id=474d5f03aba84dbaae5f5c96527310cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:13:10.689070 26691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2526aff06f5c4e70ba0ad488789723c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:10.691910 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2526aff06f5c4e70ba0ad488789723c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:10.705399 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.666021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.14.209.131:39655)
0620 06:13:10.667981 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.667991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.668267 (+ 276us) server_negotiation.cc:407] Connection header received
0620 06:13:10.668340 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.668345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.668529 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.668673 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:10.673253 (+ 4580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.674391 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.677279 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.678053 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.678166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.678289 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.682089 (+ 3800us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.682190 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1872,"thread_start_us":115,"threads_started":1}
I20260620 06:13:10.709070 26827 tablet_service.cc:1467] Processing CreateTablet for tablet 2526aff06f5c4e70ba0ad488789723c8 (DEFAULT_TABLE table=test-table [id=474d5f03aba84dbaae5f5c96527310cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:13:10.710007 26827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2526aff06f5c4e70ba0ad488789723c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:10.717653 26915 tablet_bootstrap.cc:492] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: Bootstrap starting.
I20260620 06:13:10.719941 26915 tablet_bootstrap.cc:654] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:10.720765 26915 log.cc:824] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:10.721434 26914 tablet_bootstrap.cc:492] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Bootstrap starting.
I20260620 06:13:10.723413 26914 tablet_bootstrap.cc:654] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:10.724586 26914 log.cc:824] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:10.726549 26915 tablet_bootstrap.cc:492] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: No bootstrap required, opened a new log
I20260620 06:13:10.726737 26915 ts_tablet_manager.cc:1397] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 06:13:10.732672 26914 tablet_bootstrap.cc:492] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: No bootstrap required, opened a new log
I20260620 06:13:10.732842 26914 ts_tablet_manager.cc:1397] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260620 06:13:10.737273 26914 raft_consensus.cc:348] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.737540 26914 raft_consensus.cc:374] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:10.737593 26914 raft_consensus.cc:729] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f40d69ee90644899b666c870536d89c, State: Initialized, Role: FOLLOWER
I20260620 06:13:10.738152 26916 tablet_bootstrap.cc:492] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Bootstrap starting.
I20260620 06:13:10.739212 26915 raft_consensus.cc:348] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.739459 26915 raft_consensus.cc:374] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:10.739514 26915 raft_consensus.cc:729] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e340a9f2fdb447e4b747301b280ed359, State: Initialized, Role: FOLLOWER
I20260620 06:13:10.739974 26915 consensus_queue.cc:260] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.740935 26915 ts_tablet_manager.cc:1428] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260620 06:13:10.741281 26625 heartbeater.cc:503] Master 127.14.209.190:39093 was elected leader, sending a full tablet report...
I20260620 06:13:10.742934 26916 tablet_bootstrap.cc:654] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:10.743734 26916 log.cc:824] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:10.744379 26914 consensus_queue.cc:260] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.745445 26914 ts_tablet_manager.cc:1428] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260620 06:13:10.745767 26761 heartbeater.cc:503] Master 127.14.209.190:39093 was elected leader, sending a full tablet report...
I20260620 06:13:10.747462 26916 tablet_bootstrap.cc:492] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: No bootstrap required, opened a new log
I20260620 06:13:10.747648 26916 ts_tablet_manager.cc:1397] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 06:13:10.754591 26916 raft_consensus.cc:348] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.754843 26916 raft_consensus.cc:374] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:10.754928 26916 raft_consensus.cc:729] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afc8b0034ab4582b288cb3a4249c284, State: Initialized, Role: FOLLOWER
I20260620 06:13:10.755416 26916 consensus_queue.cc:260] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.756268 26916 ts_tablet_manager.cc:1428] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:13:10.756572 26895 heartbeater.cc:503] Master 127.14.209.190:39093 was elected leader, sending a full tablet report...
I20260620 06:13:10.774716 26922 raft_consensus.cc:482] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:13:10.774883 26922 raft_consensus.cc:504] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.776311 26922 leader_election.cc:288] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e340a9f2fdb447e4b747301b280ed359 (127.14.209.129:43257), 2f40d69ee90644899b666c870536d89c (127.14.209.130:37135)
I20260620 06:13:10.780726 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.776178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:37135 (local address 127.14.209.131:37657)
0620 06:13:10.776301 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.776318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.776407 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.777007 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.777010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.777031 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:10.777150 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.777156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.778209 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.778213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.779415 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.779424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.780352 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.780355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.780451 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.780472 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.780541 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.780601 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 06:13:10.781320 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.776535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:37657 (local address 127.14.209.130:37135)
0620 06:13:10.776650 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.776657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.776673 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:13:10.776722 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.776725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.776773 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.776848 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:10.777305 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.778071 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.779562 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.780144 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.780251 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.780301 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.781120 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.781201 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:13:10.781940 26715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2526aff06f5c4e70ba0ad488789723c8" candidate_uuid: "5afc8b0034ab4582b288cb3a4249c284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f40d69ee90644899b666c870536d89c" is_pre_election: true
I20260620 06:13:10.782199 26715 raft_consensus.cc:2454] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5afc8b0034ab4582b288cb3a4249c284 in term 0.
I20260620 06:13:10.782577 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.776349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:34059 (local address 127.14.209.129:43257)
0620 06:13:10.776504 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.776511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.778424 (+ 1913us) server_negotiation.cc:407] Connection header received
0620 06:13:10.778612 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.778616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.778660 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.778706 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:10.779112 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.779826 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.781266 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.781867 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.781956 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.782003 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.782374 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.782461 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:13:10.782636 26783 leader_election.cc:302] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f40d69ee90644899b666c870536d89c, 5afc8b0034ab4582b288cb3a4249c284; no voters:
I20260620 06:13:10.782982 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.776178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:43257 (local address 127.14.209.131:34059)
0620 06:13:10.778327 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.778340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:10.778495 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:10.778819 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:10.778822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:10.778836 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:10.778932 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.778937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.779978 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.779981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.781143 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:10.781152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.782114 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.782118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:10.782212 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.782242 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:13:10.782738 (+ 496us) client_negotiation.cc:240] Negotiation successful
0620 06:13:10.782807 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"mutex_wait_us":369,"thread_start_us":99,"threads_started":1}
I20260620 06:13:10.783563 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2526aff06f5c4e70ba0ad488789723c8" candidate_uuid: "5afc8b0034ab4582b288cb3a4249c284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e340a9f2fdb447e4b747301b280ed359" is_pre_election: true
I20260620 06:13:10.783810 26565 raft_consensus.cc:2454] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5afc8b0034ab4582b288cb3a4249c284 in term 0.
I20260620 06:13:10.783262 26922 raft_consensus.cc:2781] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:13:10.784518 26922 raft_consensus.cc:482] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:13:10.784617 26922 raft_consensus.cc:3037] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:10.785948 26922 raft_consensus.cc:504] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.786556 26715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2526aff06f5c4e70ba0ad488789723c8" candidate_uuid: "5afc8b0034ab4582b288cb3a4249c284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f40d69ee90644899b666c870536d89c"
I20260620 06:13:10.786676 26715 raft_consensus.cc:3037] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:10.787755 26715 raft_consensus.cc:2454] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5afc8b0034ab4582b288cb3a4249c284 in term 1.
I20260620 06:13:10.788005 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2526aff06f5c4e70ba0ad488789723c8" candidate_uuid: "5afc8b0034ab4582b288cb3a4249c284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e340a9f2fdb447e4b747301b280ed359"
I20260620 06:13:10.788125 26565 raft_consensus.cc:3037] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:10.789376 26565 raft_consensus.cc:2454] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5afc8b0034ab4582b288cb3a4249c284 in term 1.
I20260620 06:13:10.790169 26783 leader_election.cc:302] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f40d69ee90644899b666c870536d89c, 5afc8b0034ab4582b288cb3a4249c284; no voters:
I20260620 06:13:10.790831 26922 leader_election.cc:288] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [CANDIDATE]: Term 1 election: Requested vote from peers e340a9f2fdb447e4b747301b280ed359 (127.14.209.129:43257), 2f40d69ee90644899b666c870536d89c (127.14.209.130:37135)
I20260620 06:13:10.790990 26922 raft_consensus.cc:2781] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:13:10.791095 26922 raft_consensus.cc:686] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 1 LEADER]: Becoming Leader. State: Replica: 5afc8b0034ab4582b288cb3a4249c284, State: Running, Role: LEADER
I20260620 06:13:10.791401 26922 consensus_queue.cc:237] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } }
I20260620 06:13:10.794446 26411 catalog_manager.cc:5495] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5afc8b0034ab4582b288cb3a4249c284 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "5afc8b0034ab4582b288cb3a4249c284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5afc8b0034ab4582b288cb3a4249c284" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 39655 } health_report { overall_health: HEALTHY } } }
Jun 20 06:13:10 dist-test-slave-zt01 krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:10.830188 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.825224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.14.209.131:39655)
0620 06:13:10.825386 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:10.825391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:10.825408 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:13:10.825571 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:10.825574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:10.825644 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:10.825754 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:13:10.826151 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.826949 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.828270 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:10.828601 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:10.828713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:10.829153 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:13:10.829531 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:13:10.829669 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:10.830025 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 06:13:10.830084 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:13:10.839818 26715 raft_consensus.cc:1261] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 1 FOLLOWER]: Refusing update from remote peer 5afc8b0034ab4582b288cb3a4249c284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:13:10.839818 26565 raft_consensus.cc:1261] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 1 FOLLOWER]: Refusing update from remote peer 5afc8b0034ab4582b288cb3a4249c284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:13:10.840509 26924 consensus_queue.cc:1035] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e340a9f2fdb447e4b747301b280ed359" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:13:10.840716 26922 consensus_queue.cc:1035] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f40d69ee90644899b666c870536d89c" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 37135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:13:10.856726 26930 mvcc.cc:205] Tried to move back new op lower bound from 7298809818472161280 to 7298809818284466176. Current Snapshot: MvccSnapshot[applied={T|T < 7298809818472161280}]
I20260620 06:13:10.857504 26928 mvcc.cc:205] Tried to move back new op lower bound from 7298809818472161280 to 7298809818284466176. Current Snapshot: MvccSnapshot[applied={T|T < 7298809818472161280}]
I20260620 06:13:10.861927 26929 mvcc.cc:205] Tried to move back new op lower bound from 7298809818472161280 to 7298809818284466176. Current Snapshot: MvccSnapshot[applied={T|T < 7298809818472161280}]
I20260620 06:13:10.870132 26410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:13:10.870281 26410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:13:10.872740 26410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e28566a69dd4ea9897a5d711370233d: Sending DeleteTablet for 3 replicas of tablet 2526aff06f5c4e70ba0ad488789723c8
I20260620 06:13:10.873493 26559 tablet_service.cc:1514] Processing DeleteTablet for tablet 2526aff06f5c4e70ba0ad488789723c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:13:10 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:38484
I20260620 06:13:10.873716 26691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2526aff06f5c4e70ba0ad488789723c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:13:10 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:43636
I20260620 06:13:10.873929 26938 tablet_replica.cc:330] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: stopping tablet replica
I20260620 06:13:10.874185 26938 raft_consensus.cc:2229] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:13:10.874440 26938 raft_consensus.cc:2258] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:13:10.874563 26827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2526aff06f5c4e70ba0ad488789723c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:13:10 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:48512
I20260620 06:13:10.874858 26939 tablet_replica.cc:330] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: stopping tablet replica
I20260620 06:13:10.875141 26939 raft_consensus.cc:2229] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:13:10.875393 26939 raft_consensus.cc:2258] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:13:10.876039 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 26492
I20260620 06:13:10.876405 26938 ts_tablet_manager.cc:1905] T 2526aff06f5c4e70ba0ad488789723c8 P e340a9f2fdb447e4b747301b280ed359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:13:10.879244 26940 tablet_replica.cc:330] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: stopping tablet replica
I20260620 06:13:10.879482 26940 raft_consensus.cc:2229] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:13:10.879818 26940 raft_consensus.cc:2258] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:13:10.882037 26939 ts_tablet_manager.cc:1905] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:13:10.886960 26939 ts_tablet_manager.cc:1918] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:13:10.890244 26939 log.cc:1196] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/2526aff06f5c4e70ba0ad488789723c8
I20260620 06:13:10.890659 26939 ts_tablet_manager.cc:1939] T 2526aff06f5c4e70ba0ad488789723c8 P 2f40d69ee90644899b666c870536d89c: Deleting consensus metadata
I20260620 06:13:10.892109 26940 ts_tablet_manager.cc:1905] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:13:10.895108 26940 ts_tablet_manager.cc:1918] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:13:10.895306 26940 log.cc:1196] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/2526aff06f5c4e70ba0ad488789723c8
I20260620 06:13:10.895701 26940 ts_tablet_manager.cc:1939] T 2526aff06f5c4e70ba0ad488789723c8 P 5afc8b0034ab4582b288cb3a4249c284: Deleting consensus metadata
I20260620 06:13:10.896800 26397 catalog_manager.cc:4841] TS 5afc8b0034ab4582b288cb3a4249c284 (127.14.209.131:39655): tablet 2526aff06f5c4e70ba0ad488789723c8 (table test-table [id=474d5f03aba84dbaae5f5c96527310cd]) successfully deleted
I20260620 06:13:10.897812 26396 catalog_manager.cc:4841] TS 2f40d69ee90644899b666c870536d89c (127.14.209.130:37135): tablet 2526aff06f5c4e70ba0ad488789723c8 (table test-table [id=474d5f03aba84dbaae5f5c96527310cd]) successfully deleted
W20260620 06:13:10.898313 26397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:43257 (error 108)
I20260620 06:13:10.899747 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:10.899455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:43257 (local address 127.0.0.1:38488)
0620 06:13:10.899611 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:10.899671 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:43257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 06:13:10.899986 26397 catalog_manager.cc:4584] TS e340a9f2fdb447e4b747301b280ed359 (127.14.209.129:43257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2526aff06f5c4e70ba0ad488789723c8: Network error: Client connection negotiation failed: client connection to 127.14.209.129:43257: connect: Connection refused (error 111)
I20260620 06:13:10.900317 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 26631
I20260620 06:13:10.909071 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 26766
I20260620 06:13:10.917292 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 26379
I20260620 06:13:10.925599 15174 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:13:11.269915 15174 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:13:11.270030 15174 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:13:11.614193 15174 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:13:11.614 UTC [25965] LOG: received smart shutdown request
2026-06-20 06:13:11.617 UTC [25965] LOG: background worker "logical replication launcher" (PID 25974) exited with exit code 1
2026-06-20 06:13:11.617 UTC [25969] LOG: shutting down
2026-06-20 06:13:11.775 UTC [25965] LOG: database system is shut down
2026-06-20T06:13:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45055 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:13:11 dist-test-slave-zt01 krb5kdc[26950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:13:11 dist-test-slave-zt01 krb5kdc[26950](info): set up 2 sockets
Jun 20 06:13:11 dist-test-slave-zt01 krb5kdc[26950](info): commencing operation
krb5kdc: starting...
W20260620 06:13:12.930979 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:13:13 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:13:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/rangeradmin_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/rangeradmin_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/rangerlookup_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/rangerlookup_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
I20260620 06:13:13.089962 15174 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskepqMJ0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/postgres -l logfile start
2026-06-20 06:13:15.669 UTC [26980] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:13:15.669 UTC [26980] LOG: listening on IPv4 address "127.14.209.148", port 42351
2026-06-20 06:13:15.672 UTC [26980] LOG: listening on Unix socket "/tmp/.s.PGSQL.42351"
2026-06-20 06:13:15.702 UTC [26983] LOG: database system was shut down at 2026-06-20 06:13:14 UTC
2026-06-20 06:13:15.705 UTC [26980] LOG: database system is ready to accept connections
I20260620 06:13:16.706859 15174 mini_postgres.cc:96] Postgres bound to 42351
2026-06-20 06:13:16.713 UTC [26991] FATAL: database "slave" does not exist
127.14.209.148:42351 - accepting connections
I20260620 06:13:16.713907 15174 mini_ranger.cc:161] Starting Ranger...
I20260620 06:13:16.728159 15174 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:13:17.221365 15174 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:13:17.221451 15174 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-admin
2026-06-20 06:13:17,494 [I] DB FLAVOR :POSTGRES
2026-06-20 06:13:17,495 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:13:17,495 [I] Checking connection..
2026-06-20 06:13:17,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:13:17,737 [I] Checking connection passed.
2026-06-20 06:13:17,737 [I] --------- Verifying version history table ---------
2026-06-20 06:13:17,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:13:17,997 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:13:17,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:13:18,257 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:13:18,257 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:13:18,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:13:18.505 UTC [27055] WARNING: there is no transaction in progress
2026-06-20 06:13:18,511 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:13:18,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:13:18,732 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:13:18,732 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:13:18,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:13:18,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:13:19,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zt01.c.gcp-upstream.internal','N') ;"
2026-06-20 06:13:19,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:13:19,698 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:13:19,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:13:19,949 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:13:19,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:13:20,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:13:20,448 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:13:20,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:13:21.393 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.408 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.424 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.440 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.459 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.479 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.543 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.549 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.563 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.573 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.584 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.598 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.608 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.614 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.624 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:21.631 UTC [27163] WARNING: there is no transaction in progress
2026-06-20 06:13:22,259 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:13:22,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:13:22,469 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:13:22,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:13:22,688 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:13:22,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:13:22,937 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:13:22,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:13:23,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:13:23,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zt01.c.gcp-upstream.internal';"
2026-06-20 06:13:23,675 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:13:23,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:13:23,926 [I] DB_PATCHES have already been applied
I20260620 06:13:23.932432 15174 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-admin:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskepqMJ0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:13:23.932492 15174 mini_ranger.cc:193] Using host: 127.14.209.148
I20260620 06:13:23.935802 15174 mini_ranger.cc:227] Ranger admin URL: http://127.14.209.148:40641
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:13:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:13:25.041181 15174 mini_ranger.cc:160] Time spent starting Ranger: real 8.327s user 0.002s sys 0.006s
Jun 20, 2026 6:13:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:13:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:13:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:13:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:13:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:13:45.919282 15174 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.209.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.209.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/rangerkms_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.209.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/rangerkms_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.209.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.209.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.209.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/HTTP_127.14.209.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:13:45 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936025, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:13:46.002147 15174 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:13:46.011487 15174 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:13:46.920774 15174 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:13:46.920858 15174 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms
I20260620 06:13:46.920881 15174 mini_ranger_kms.cc:227] Using postgres at 127.14.209.148:42351
2026-06-20 06:13:47,088 [I] DB FLAVOR :POSTGRES
2026-06-20 06:13:47,088 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:13:47,088 [I] Checking connection
2026-06-20 06:13:47,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:13:47,329 [I] connection success
2026-06-20 06:13:47,329 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:13:47,329 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:13:47,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:13:47,548 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:13:47,548 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:13:47,548 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:13:47,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.209.148:42351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:13:47,786 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:13:48.245968 15174 mini_ranger_kms.cc:310] Created kms service
I20260620 06:13:48.440312 15174 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:13:48.484051 15174 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:13:48 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936025, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:48.884574 15174 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:13:48.884806 15174 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms:/tmp/dist-test-taskepqMJ0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskepqMJ0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:13:48.884836 15174 mini_ranger_kms.cc:240] Using host: 127.14.209.148
I20260620 06:13:48.888509 15174 mini_ranger_kms.cc:276] Ranger KMS PID: 27343
I20260620 06:13:48.888583 15174 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.209.148:35845
Jun 20, 2026 6:13:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:13:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 06:13:51.254640 15174 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.252s user 0.026s sys 0.125s
I20260620 06:13:51.254768 15174 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:13:51.254842 15174 mini_ranger_kms.cc:401] 127.14.209.148:35845/kms/v1/keys
Jun 20 06:13:51 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781936031, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.209.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 06:13:51 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936031, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.209.148@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
Jun 20 06:13:52 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936031, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.209.148@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
Jun 20 06:13:54 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:13:54.343439 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:35759
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:35759
--ranger_config_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:54.446547 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:54.446779 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:54.446815 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:54.450778 27394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:13:54.450827 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:54.450846 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:54.450886 27394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:13:54.450923 27394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:13:54.456018 27394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.209.148:35845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:35759
--ranger_config_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:54.456992 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:54.457930 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:13:54.466037 27394 server_base.cc:1034] running on GCE node
W20260620 06:13:54.466032 27399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:54.466040 27402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:54.466056 27400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:54.466717 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:54.467758 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:54.468922 27394 hybrid_clock.cc:648] HybridClock initialized: now 1781936034468900 us; error 34 us; skew 500 ppm
Jun 20 06:13:54 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:54.478829 27394 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:13:54.479628 27394 webserver.cc:460] Webserver started at http://127.14.209.190:44493/ using document root <none> and password file <none>
I20260620 06:13:54.480082 27394 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:54.480132 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:54.480288 27394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:54 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:54.532799 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "cd9905f24bc44f60beac694c7af57114"
format_stamp: "Formatted at 2026-06-20 06:13:54 on dist-test-slave-zt01"
server_key: "1424199bd3483e24422d43ddfab4feb3"
server_key_iv: "575fc7bae354b7064724c02548574d5f"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:54.533471 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "cd9905f24bc44f60beac694c7af57114"
format_stamp: "Formatted at 2026-06-20 06:13:54 on dist-test-slave-zt01"
server_key: "1424199bd3483e24422d43ddfab4feb3"
server_key_iv: "575fc7bae354b7064724c02548574d5f"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:54.537995 27394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:13:54.553676 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:54.555179 27394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:13:54.555320 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "cd9905f24bc44f60beac694c7af57114"
format_stamp: "Formatted at 2026-06-20 06:13:54 on dist-test-slave-zt01"
server_key: "1424199bd3483e24422d43ddfab4feb3"
server_key_iv: "575fc7bae354b7064724c02548574d5f"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:54.555434 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:54.580837 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:54.583169 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:54.583308 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:54.590206 27394 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:35759
I20260620 06:13:54.590219 27461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:35759 every 8 connection(s)
I20260620 06:13:54.591084 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:13:54.593523 27462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:54.598397 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Bootstrap starting.
I20260620 06:13:54.599514 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 27394
I20260620 06:13:54.599635 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:13:54.599892 15174 external_mini_cluster.cc:1428] Setting key 3e0e33b1f962140e680769f7d09ed499
I20260620 06:13:54.600797 27462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:54.601512 27462 log.cc:824] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:54.603515 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: No bootstrap required, opened a new log
I20260620 06:13:54.607568 27462 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:54.607831 27462 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:54.607895 27462 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9905f24bc44f60beac694c7af57114, State: Initialized, Role: FOLLOWER
I20260620 06:13:54.608361 27462 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
Jun 20 06:13:54 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:54.608505 27462 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:13:54.608603 27462 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:13:54.608707 27462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:54.609575 27462 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:54.609877 27462 leader_election.cc:302] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9905f24bc44f60beac694c7af57114; no voters:
I20260620 06:13:54.610158 27462 leader_election.cc:288] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:13:54.610234 27467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:13:54.610472 27467 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 LEADER]: Becoming Leader. State: Replica: cd9905f24bc44f60beac694c7af57114, State: Running, Role: LEADER
I20260620 06:13:54.610793 27467 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:54.611362 27462 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:13:54.612432 27468 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9905f24bc44f60beac694c7af57114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } } }
I20260620 06:13:54.612542 27468 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: This master's current role is: LEADER
I20260620 06:13:54.612907 27469 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9905f24bc44f60beac694c7af57114. Latest consensus state: current_term: 1 leader_uuid: "cd9905f24bc44f60beac694c7af57114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } } }
I20260620 06:13:54.613054 27469 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: This master's current role is: LEADER
I20260620 06:13:54.613474 27462 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:13:54.613934 27476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:13:54.614473 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:54.600865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.14.209.190:35759)
0620 06:13:54.601245 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:54.601260 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:54.601293 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:13:54.601999 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:54.602021 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:54.602357 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:54.602610 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:54.603503 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:54.604457 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:54.605720 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:54.606135 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:54.606249 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:54.609105 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:54.609110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:54.609120 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:54.609143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:54.611022 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:54.611538 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:54.611543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:54.611549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:54.611693 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:54.612025 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:54.612028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:54.612030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:54.612556 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:54.612741 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:54.613442 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 06:13:54.613713 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260620 06:13:54.619868 27476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:13:54.625869 27476 catalog_manager.cc:1347] Generated new cluster ID: dc4c4e47f3824d04accd9d8fa86e6e64
I20260620 06:13:54.625922 27476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:13:54.637113 27476 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:13:54.638327 27476 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:13:54.644384 27476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Generated new TSK 0
I20260620 06:13:54.645025 27476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:13:55.301731 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:35759
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:55.416857 27507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:55.417079 27507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:55.417119 27507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:55.421186 27507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:55.421238 27507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:55.421320 27507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:13:55.427162 27507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.209.148:35845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:55.428155 27507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:55.429109 27507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:55.438643 27513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:55.438630 27512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:55.438961 27515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:55.438993 27507 server_base.cc:1034] running on GCE node
I20260620 06:13:55.439458 27507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:55.440174 27507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:55.441372 27507 hybrid_clock.cc:648] HybridClock initialized: now 1781936035441361 us; error 28 us; skew 500 ppm
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:55.444586 27507 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:13:55.445422 27507 webserver.cc:460] Webserver started at http://127.14.209.129:37311/ using document root <none> and password file <none>
I20260620 06:13:55.445999 27507 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:55.446089 27507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:55.446295 27507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:55.460630 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "38160083e02147af88f9cf8e3bb0b90e"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "65d6902fa27cc9e10788c6c2237ff556"
server_key_iv: "27f146599496c4b31d63f37fe72f7039"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:55.461506 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "38160083e02147af88f9cf8e3bb0b90e"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "65d6902fa27cc9e10788c6c2237ff556"
server_key_iv: "27f146599496c4b31d63f37fe72f7039"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:55.468257 27507 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 06:13:55.487848 27524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:55.489385 27507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:13:55.489495 27507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "38160083e02147af88f9cf8e3bb0b90e"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "65d6902fa27cc9e10788c6c2237ff556"
server_key_iv: "27f146599496c4b31d63f37fe72f7039"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:55.489606 27507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:55.515012 27507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:55.517863 27507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:55.518051 27507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:55.518622 27507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:55.519436 27507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:55.519500 27507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:55.519569 27507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:55.519601 27507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:55.529786 27507 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:45447
I20260620 06:13:55.529940 27637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:45447 every 8 connection(s)
I20260620 06:13:55.530743 27507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:13:55.538547 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 27507
I20260620 06:13:55.538681 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:13:55.538945 15174 external_mini_cluster.cc:1428] Setting key 4ffcba058856e3cb2da2ece80955df7c
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:55.545482 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:55.532666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:36085 (local address 127.14.209.190:35759)
0620 06:13:55.532931 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:55.532939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:55.533633 (+ 694us) server_negotiation.cc:407] Connection header received
0620 06:13:55.534564 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:55.534567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:55.534626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:55.534709 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:55.535747 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.536304 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.537362 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.537657 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.537759 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:55.540988 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:55.540995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:55.541000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:55.541030 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:55.542959 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:55.543631 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:55.543637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:55.543642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:55.543735 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:55.544141 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:55.544147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:55.544152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:55.544384 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:55.544466 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:55.545145 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 06:13:55.545278 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260620 06:13:55.546669 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:55.533078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35759 (local address 127.14.209.129:36085)
0620 06:13:55.533498 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:55.533544 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:55.534366 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:55.534861 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:55.534868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:55.535163 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:55.535609 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:55.535621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.536422 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.536425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:55.537236 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:55.537245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.537789 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.537793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:55.537952 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:55.539169 (+ 1217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:55.539191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:55.540801 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:55.543133 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:55.543139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:55.543154 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:55.543494 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:55.543847 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:55.543852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:55.543857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:55.544035 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:55.544536 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:55.544544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:55.544831 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 06:13:55.545827 (+ 996us) client_negotiation.cc:240] Negotiation successful
0620 06:13:55.546046 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
I20260620 06:13:55.547715 27638 heartbeater.cc:347] Connected to a master server at 127.14.209.190:35759
I20260620 06:13:55.547915 27638 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:55.548327 27638 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:55.549628 27426 ts_manager.cc:194] Registered new tserver with Master: 38160083e02147af88f9cf8e3bb0b90e (127.14.209.129:45447)
I20260620 06:13:55.550657 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:36085
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:13:55.581434 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:35759
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 06:13:55.746505 27646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:55.746851 27646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:55.746948 27646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:55.752907 27646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:55.753012 27646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:55.753132 27646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:13:55.760772 27646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.209.148:35845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:55.762063 27646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:55.763356 27646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:55.775084 27653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:55.775149 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:55.775365 27655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:55.775595 27646 server_base.cc:1034] running on GCE node
I20260620 06:13:55.776010 27646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:55.776613 27646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:55.777781 27646 hybrid_clock.cc:648] HybridClock initialized: now 1781936035777765 us; error 34 us; skew 500 ppm
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:55.780941 27646 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:13:55.781725 27646 webserver.cc:460] Webserver started at http://127.14.209.130:46863/ using document root <none> and password file <none>
I20260620 06:13:55.782248 27646 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:55.782330 27646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:55.782583 27646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:55.795111 27646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "2e6544f55dd24bc69e84ecdf9fe91f52"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "0bf4aa01c90ce177de0eb16b186307ed"
server_key_iv: "fe04233c711f08b5dc9c4ceffc6786e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:55.795709 27646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "2e6544f55dd24bc69e84ecdf9fe91f52"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "0bf4aa01c90ce177de0eb16b186307ed"
server_key_iv: "fe04233c711f08b5dc9c4ceffc6786e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:55.799273 27646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:13:55.827415 27662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:55.829077 27646 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260620 06:13:55.829267 27646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "2e6544f55dd24bc69e84ecdf9fe91f52"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "0bf4aa01c90ce177de0eb16b186307ed"
server_key_iv: "fe04233c711f08b5dc9c4ceffc6786e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:55.829478 27646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:55.850445 27646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:55.853863 27646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:55.854086 27646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:55.854756 27646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:55.855831 27646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:55.855934 27646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:55.856056 27646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:55.856127 27646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:55.866721 27646 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:36769
I20260620 06:13:55.867861 27646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:13:55.872017 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 27646
I20260620 06:13:55.872150 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:13:55.872499 15174 external_mini_cluster.cc:1428] Setting key 21de802be326cb5df4249b4132492dc7
I20260620 06:13:55.875407 27775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:36769 every 8 connection(s)
Jun 20 06:13:55 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:55.891995 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:55.879381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:45311 (local address 127.14.209.190:35759)
0620 06:13:55.879514 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:55.879519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:55.879535 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:13:55.880580 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:55.880583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:55.880631 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:55.880693 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:55.882196 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.883088 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.884737 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.885105 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.885173 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:55.888126 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:55.888130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:55.888133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:55.888157 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:55.889814 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:55.890372 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:55.890375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:55.890377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:55.890467 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:55.890827 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:55.890830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:55.890832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:55.891026 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:55.891115 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:55.891713 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 06:13:55.891832 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:13:55.893612 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:55.878242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35759 (local address 127.14.209.130:45311)
0620 06:13:55.879347 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:55.879398 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:55.880338 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:55.881085 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:55.881094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:55.881420 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:55.882035 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:55.882052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.883234 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.883238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:55.884603 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:55.884613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:55.885354 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:55.885358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:55.885556 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:55.886226 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:55.886245 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:55.887952 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:55.889940 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:55.889946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:55.889959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:55.890261 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:55.890573 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:55.890575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:55.890577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:55.890728 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:55.891191 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:55.891197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:55.891462 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 06:13:55.892395 (+ 933us) client_negotiation.cc:240] Negotiation successful
0620 06:13:55.892693 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":147,"threads_started":1}
I20260620 06:13:55.894955 27776 heartbeater.cc:347] Connected to a master server at 127.14.209.190:35759
I20260620 06:13:55.895226 27776 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:55.895797 27776 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:55.896800 27426 ts_manager.cc:194] Registered new tserver with Master: 2e6544f55dd24bc69e84ecdf9fe91f52 (127.14.209.130:36769)
I20260620 06:13:55.897503 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:45311
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:13:55.930867 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:35759
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:56.074427 27781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:56.074805 27781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:56.074924 27781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:56.081557 27781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:56.081624 27781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:56.081708 27781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:13:56.089147 27781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.209.148:35845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:56.090287 27781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:56.091528 27781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:56.103255 27789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:56.103243 27787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:56.103507 27786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:56.104636 27781 server_base.cc:1034] running on GCE node
I20260620 06:13:56.105400 27781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:56.106565 27781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:56.110810 27781 hybrid_clock.cc:648] HybridClock initialized: now 1781936036110723 us; error 116 us; skew 500 ppm
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:56.114063 27781 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:13:56.115089 27781 webserver.cc:460] Webserver started at http://127.14.209.131:32991/ using document root <none> and password file <none>
I20260620 06:13:56.115840 27781 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:56.115990 27781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:13:56.116277 27781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936035, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:56.149794 27781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "fa9e523a71c84da1bf831fdf65a26244"
format_stamp: "Formatted at 2026-06-20 06:13:56 on dist-test-slave-zt01"
server_key: "59c8051466b09f14579eb4a793b64ce2"
server_key_iv: "20af38a0b496d8507a1232425b42cf40"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:56.150458 27781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "fa9e523a71c84da1bf831fdf65a26244"
format_stamp: "Formatted at 2026-06-20 06:13:56 on dist-test-slave-zt01"
server_key: "59c8051466b09f14579eb4a793b64ce2"
server_key_iv: "20af38a0b496d8507a1232425b42cf40"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:56.155484 27781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 06:13:56.174755 27796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.177987 27781 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260620 06:13:56.178176 27781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "fa9e523a71c84da1bf831fdf65a26244"
format_stamp: "Formatted at 2026-06-20 06:13:56 on dist-test-slave-zt01"
server_key: "59c8051466b09f14579eb4a793b64ce2"
server_key_iv: "20af38a0b496d8507a1232425b42cf40"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:56.178426 27781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:56.206241 27781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:56.209770 27781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:56.210064 27781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:56.211136 27781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:56.212158 27781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:56.212251 27781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.212335 27781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:56.212389 27781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.223019 27781 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:41737
I20260620 06:13:56.223263 27909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:41737 every 8 connection(s)
I20260620 06:13:56.224207 27781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:13:56.230796 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 27781
I20260620 06:13:56.230942 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:13:56.231269 15174 external_mini_cluster.cc:1428] Setting key 73e22f3e4c9ab53e7db49e8db99c66c8
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:56.240247 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:56.226097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:41275 (local address 127.14.209.190:35759)
0620 06:13:56.226243 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:56.226247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:56.227212 (+ 965us) server_negotiation.cc:407] Connection header received
0620 06:13:56.228395 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:56.228401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:56.228457 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:56.228555 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:56.229819 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.230676 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.232286 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.232639 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.232716 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:56.236902 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:56.236906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:56.236909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:56.236937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:56.238271 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:56.238764 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:56.238772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:56.238773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:56.238844 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:56.239219 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:56.239223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:56.239224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:56.239393 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:56.239497 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:56.239958 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 06:13:56.240087 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:13:56.240985 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:56.226521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35759 (local address 127.14.209.131:41275)
0620 06:13:56.227044 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:56.227119 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:56.228158 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:56.228728 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:56.228736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:56.229043 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:56.229640 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:56.229661 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.230835 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.230839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:56.232159 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:56.232169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.232901 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.232905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:56.233089 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:56.235216 (+ 2127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:56.235236 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:56.236743 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:56.238386 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:56.238391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:56.238404 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:56.238658 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:56.238932 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:56.238937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:56.238939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:56.239120 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:56.239501 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:56.239509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:56.239734 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 06:13:56.239902 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 06:13:56.240180 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":141,"threads_started":1}
I20260620 06:13:56.241986 27910 heartbeater.cc:347] Connected to a master server at 127.14.209.190:35759
I20260620 06:13:56.242218 27910 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:56.242650 27910 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:56.243638 27426 ts_manager.cc:194] Registered new tserver with Master: fa9e523a71c84da1bf831fdf65a26244 (127.14.209.131:41737)
I20260620 06:13:56.244377 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:41275
I20260620 06:13:56.244772 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:13:56.245325 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 27507
I20260620 06:13:56.251602 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 27646
I20260620 06:13:56.259299 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 27781
I20260620 06:13:56.264786 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 27394
I20260620 06:13:56.273018 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:35759
--webserver_interface=127.14.209.190
--webserver_port=44493
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:35759
--ranger_config_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:13:56.374979 27913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:56.375238 27913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:56.375316 27913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:56.379220 27913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:13:56.379292 27913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:56.379334 27913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:56.379383 27913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:13:56.379421 27913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:13:56.384999 27913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:35759
--ranger_config_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=44493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:56.385861 27913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:56.386719 27913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:56.394433 27918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:56.394497 27913 server_base.cc:1034] running on GCE node
W20260620 06:13:56.394452 27919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:56.394434 27921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:56.395146 27913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:56.396108 27913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:56.397279 27913 hybrid_clock.cc:648] HybridClock initialized: now 1781936036397254 us; error 38 us; skew 500 ppm
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:56.399840 27913 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:13:56.400583 27913 webserver.cc:460] Webserver started at http://127.14.209.190:44493/ using document root <none> and password file <none>
I20260620 06:13:56.401014 27913 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:56.401072 27913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:56.481413 27913 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:13:56.483670 27929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.484819 27913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:13:56.484941 27913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "cd9905f24bc44f60beac694c7af57114"
format_stamp: "Formatted at 2026-06-20 06:13:54 on dist-test-slave-zt01"
server_key: "1424199bd3483e24422d43ddfab4feb3"
server_key_iv: "575fc7bae354b7064724c02548574d5f"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:56.485431 27913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:56.503628 27913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:56.506281 27913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:56.506421 27913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:56.513581 27913 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:35759
I20260620 06:13:56.513617 27981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:35759 every 8 connection(s)
I20260620 06:13:56.514539 27913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:13:56.518359 27982 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:13:56.519258 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 27913
I20260620 06:13:56.519733 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:45447
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:35759
--webserver_port=37311
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:13:56.520203 27982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Bootstrap starting.
I20260620 06:13:56.536121 27982 log.cc:824] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:56.542032 27982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:13:56.542393 27982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Bootstrap complete.
I20260620 06:13:56.546327 27982 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:56.546677 27982 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9905f24bc44f60beac694c7af57114, State: Initialized, Role: FOLLOWER
I20260620 06:13:56.547158 27982 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:56.547297 27982 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:13:56.547389 27982 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:13:56.547495 27982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:13:56.549168 27982 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:56.549475 27982 leader_election.cc:302] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9905f24bc44f60beac694c7af57114; no voters:
I20260620 06:13:56.549739 27982 leader_election.cc:288] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:13:56.549875 27986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:13:56.550103 27986 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [term 2 LEADER]: Becoming Leader. State: Replica: cd9905f24bc44f60beac694c7af57114, State: Running, Role: LEADER
I20260620 06:13:56.550448 27986 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } }
I20260620 06:13:56.550822 27982 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:13:56.551731 27987 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd9905f24bc44f60beac694c7af57114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } } }
I20260620 06:13:56.551862 27987 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: This master's current role is: LEADER
I20260620 06:13:56.552210 27988 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9905f24bc44f60beac694c7af57114. Latest consensus state: current_term: 2 leader_uuid: "cd9905f24bc44f60beac694c7af57114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9905f24bc44f60beac694c7af57114" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35759 } } }
I20260620 06:13:56.552338 27988 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114 [sys.catalog]: This master's current role is: LEADER
I20260620 06:13:56.552635 27995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:13:56.553012 27982 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:13:56.558570 27995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:13:56.559386 27995 catalog_manager.cc:1259] Loaded cluster ID: dc4c4e47f3824d04accd9d8fa86e6e64
I20260620 06:13:56.559484 27995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:13:56.562953 27995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:13:56.564950 27995 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Loaded TSK: 0
I20260620 06:13:56.566155 27995 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:13:56.649664 27984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:56.649950 27984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:56.650012 27984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:56.656260 27984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:56.656363 27984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:56.656462 27984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:13:56.663921 27984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:45447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=37311
--tserver_master_addrs=127.14.209.190:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:56.665169 27984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:56.666370 27984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:56.677851 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:56.678485 27984 server_base.cc:1034] running on GCE node
W20260620 06:13:56.678575 28019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:56.678691 28020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:56.679214 27984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:56.679955 27984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:56.681143 27984 hybrid_clock.cc:648] HybridClock initialized: now 1781936036681113 us; error 44 us; skew 500 ppm
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:56.687361 27984 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:13:56.688759 27984 webserver.cc:460] Webserver started at http://127.14.209.129:37311/ using document root <none> and password file <none>
I20260620 06:13:56.689771 27984 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:56.689944 27984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:56.741413 27984 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260620 06:13:56.745615 28030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.747130 27984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:13:56.747295 27984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "38160083e02147af88f9cf8e3bb0b90e"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "65d6902fa27cc9e10788c6c2237ff556"
server_key_iv: "27f146599496c4b31d63f37fe72f7039"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:56.748013 27984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:56.780159 27984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:56.783586 27984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:56.783772 27984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:56.784402 27984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:56.785372 27984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:56.785444 27984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.785549 27984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:56.785588 27984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:56.795701 27984 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:45447
I20260620 06:13:56.796891 27984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:13:56.797736 28143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:45447 every 8 connection(s)
I20260620 06:13:56.806341 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 27984
I20260620 06:13:56.806792 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:36769
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:35759
--webserver_port=46863
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:56.822161 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:56.800002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35759 (local address 127.14.209.129:33857)
0620 06:13:56.800459 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:56.800507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:56.801358 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:56.804469 (+ 3111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:56.804482 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:56.804715 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:56.805189 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:56.805201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.809006 (+ 3805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.809009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:56.809826 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:56.809835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.810886 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.810890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:56.811124 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:56.812152 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:56.812172 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:56.815259 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:56.818679 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:56.818685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:56.818699 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:56.819045 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:56.819511 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:56.819514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:56.819516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:56.819678 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:56.820394 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:56.820400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:56.820671 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 06:13:56.820852 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 06:13:56.821158 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20260620 06:13:56.823855 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:56.799952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:33857 (local address 127.14.209.190:35759)
0620 06:13:56.803286 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:56.803302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:56.803335 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:13:56.803548 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:56.803558 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:56.803865 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:56.804162 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:56.805343 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.806821 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.809952 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:56.810706 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:56.812888 (+ 2182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:56.815492 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:56.815498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:56.815512 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:56.815544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:56.817627 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:56.819223 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:56.819229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:56.819234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:56.819390 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:56.819798 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:56.819801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:56.819804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:56.820247 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:56.822563 (+ 2316us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:56.822733 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:13:56.823007 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3203,"thread_start_us":107,"threads_started":1}
I20260620 06:13:56.826364 28144 heartbeater.cc:347] Connected to a master server at 127.14.209.190:35759
I20260620 06:13:56.826586 28144 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:56.827113 28144 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
W20260620 06:13:56.829242 28144 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:35759 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:13:56.829375 28144 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:56.829527 28144 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:56.830462 28144 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:56.830619 28144 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
W20260620 06:13:56.831447 28144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:13:56.947525 28148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:56.947753 28148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:56.947796 28148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:56.952435 28148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:56.952571 28148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:56.952667 28148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:13:56.959789 28148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=46863
--tserver_master_addrs=127.14.209.190:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:56.960868 28148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:56.961947 28148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:56.972666 28153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:56.972700 28156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:56.974404 28154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:56.975087 28148 server_base.cc:1034] running on GCE node
I20260620 06:13:56.975569 28148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:56.976328 28148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:56.977530 28148 hybrid_clock.cc:648] HybridClock initialized: now 1781936036977499 us; error 41 us; skew 500 ppm
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:56.981047 28148 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:13:56.982105 28148 webserver.cc:460] Webserver started at http://127.14.209.130:46863/ using document root <none> and password file <none>
I20260620 06:13:56.982767 28148 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:56.982868 28148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:13:56 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:57.007234 28148 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:13:57.011801 28164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:57.013602 28148 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 06:13:57.013721 28148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "2e6544f55dd24bc69e84ecdf9fe91f52"
format_stamp: "Formatted at 2026-06-20 06:13:55 on dist-test-slave-zt01"
server_key: "0bf4aa01c90ce177de0eb16b186307ed"
server_key_iv: "fe04233c711f08b5dc9c4ceffc6786e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:57.014402 28148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:57.031643 28148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:57.034976 28148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:57.035190 28148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:57.035832 28148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:57.036796 28148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:57.036862 28148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:57.036916 28148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:57.036947 28148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:57.047549 28148 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:36769
I20260620 06:13:57.047952 28278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:36769 every 8 connection(s)
I20260620 06:13:57.048702 28148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:13:57.058250 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28148
I20260620 06:13:57.058795 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.209.148:35845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:41737
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:35759
--webserver_port=32991
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:13:57 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936036, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:57.065404 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:57.052340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35759 (local address 127.14.209.130:37893)
0620 06:13:57.052968 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:57.053073 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:57.053969 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:57.054506 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:57.054512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:57.054757 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:57.055298 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:57.055312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.056080 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.056084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:57.056926 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:57.056934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.057433 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.057437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:57.057601 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:57.058234 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:57.058255 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:57.059996 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:57.062205 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:57.062213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:57.062229 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:57.062599 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:57.062968 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:57.062971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:57.062973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:57.063174 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:57.063668 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:57.063674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:57.063961 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 06:13:57.064157 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 06:13:57.064457 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":162,"threads_started":1}
I20260620 06:13:57.066356 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:57.051758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:37893 (local address 127.14.209.190:35759)
0620 06:13:57.051917 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:57.051923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:57.053174 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 06:13:57.054194 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:57.054197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:57.054260 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:57.054345 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:57.055430 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.055961 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.057046 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.057287 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.057398 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:57.060196 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:57.060203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:57.060206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:57.060235 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:57.062079 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:57.062714 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:57.062718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:57.062720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:57.062817 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:57.063289 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:57.063294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:57.063297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:57.063509 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:57.063612 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:57.066062 (+ 2450us) server_negotiation.cc:299] Negotiation successful
0620 06:13:57.066190 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 06:13:57.067381 28279 heartbeater.cc:347] Connected to a master server at 127.14.209.190:35759
I20260620 06:13:57.067600 28279 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:57.068063 28279 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
W20260620 06:13:57.069725 28279 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:35759 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:13:57.069845 28279 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:57.070007 28279 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:57.070895 28279 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:57.071084 28279 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
W20260620 06:13:57.072176 28279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:13:57.194905 28282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:13:57.195225 28282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:13:57.195302 28282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:13:57.200510 28282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:13:57.200562 28282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:13:57.200641 28282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:13:57.206420 28282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.209.148:35845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:41737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=32991
--tserver_master_addrs=127.14.209.190:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:13:57.207394 28282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:13:57.208317 28282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:13:57.218991 28288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:13:57.219010 28290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:57.219393 28282 server_base.cc:1034] running on GCE node
W20260620 06:13:57.219429 28287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:13:57.222455 28282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:13:57.223069 28282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:13:57.224285 28282 hybrid_clock.cc:648] HybridClock initialized: now 1781936037224253 us; error 53 us; skew 500 ppm
Jun 20 06:13:57 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:57.226931 28282 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:13:57.227910 28282 webserver.cc:460] Webserver started at http://127.14.209.131:32991/ using document root <none> and password file <none>
I20260620 06:13:57.228550 28282 fs_manager.cc:359] Metadata directory not provided
I20260620 06:13:57.228660 28282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:13:57 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:57.245330 28282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:13:57.247793 28298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:57.249046 28282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:13:57.249161 28282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "fa9e523a71c84da1bf831fdf65a26244"
format_stamp: "Formatted at 2026-06-20 06:13:56 on dist-test-slave-zt01"
server_key: "59c8051466b09f14579eb4a793b64ce2"
server_key_iv: "20af38a0b496d8507a1232425b42cf40"
server_key_version: "kuduclusterkey@0"
I20260620 06:13:57.249684 28282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:13:57.269464 28282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:13:57.272946 28282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:13:57.273144 28282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:13:57.273877 28282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:13:57.274911 28282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:13:57.274988 28282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:13:57.275045 28282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:13:57.275120 28282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 06:13:57.286343 28282 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:41737
I20260620 06:13:57.286423 28411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:41737 every 8 connection(s)
I20260620 06:13:57.287606 28282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:13:57.288775 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28282
Jun 20 06:13:57 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:13:57.300137 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:57.289497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.14.209.190:35759)
0620 06:13:57.289788 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:57.289792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:57.290017 (+ 225us) server_negotiation.cc:407] Connection header received
0620 06:13:57.290172 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:57.290178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:57.290234 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:57.290310 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:57.291443 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.292273 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.293473 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.293839 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.293948 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:57.296333 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:57.296339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:57.296342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:57.296371 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:57.298155 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:57.298817 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:57.298820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:57.298821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:57.298896 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:57.299212 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:57.299216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:57.299218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:57.299419 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:57.299613 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:57.299832 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 06:13:57.299962 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260620 06:13:57.305277 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:57.290261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:53121 (local address 127.14.209.190:35759)
0620 06:13:57.290502 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:57.290506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:57.290967 (+ 461us) server_negotiation.cc:407] Connection header received
0620 06:13:57.292028 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:57.292031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:57.292088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:57.292194 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:57.293899 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.294406 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.296368 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.296702 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.296813 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:57.301053 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:57.301055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:57.301057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:57.301102 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:57.302774 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:57.303527 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:57.303530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:57.303532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:57.303622 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:57.304017 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:57.304020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:57.304022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:57.304225 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:57.304313 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:57.304998 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 06:13:57.305115 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260620 06:13:57.306991 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:57.290255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35759 (local address 127.14.209.131:53121)
0620 06:13:57.290804 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:57.290858 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:57.291830 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:57.292917 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:57.292926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:57.293235 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:13:57.293766 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:57.293778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.294526 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.294531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:57.296197 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:57.296207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:57.297236 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:57.297243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:57.297461 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:57.298038 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:13:57.298062 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:13:57.300107 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:13:57.302991 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:57.302996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:57.303011 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:57.303408 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:57.303727 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:13:57.303730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:13:57.303732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:13:57.303891 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:13:57.304398 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:13:57.304404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:13:57.304707 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 06:13:57.305681 (+ 974us) client_negotiation.cc:240] Negotiation successful
0620 06:13:57.306014 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":145,"threads_started":1}
I20260620 06:13:57.308354 28412 heartbeater.cc:347] Connected to a master server at 127.14.209.190:35759
I20260620 06:13:57.308599 28412 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:57.309064 28412 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
W20260620 06:13:57.310628 28412 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:35759 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:13:57.310745 28412 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:57.310921 28412 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:57.311913 27946 ts_manager.cc:194] Registered new tserver with Master: fa9e523a71c84da1bf831fdf65a26244 (127.14.209.131:41737)
I20260620 06:13:57.313042 27946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:53121
Jun 20 06:13:57 dist-test-slave-zt01 krb5kdc[26950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781936037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:13:57.831660 28144 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:57.831955 28144 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:57.832767 27946 ts_manager.cc:194] Registered new tserver with Master: 38160083e02147af88f9cf8e3bb0b90e (127.14.209.129:45447)
I20260620 06:13:57.833549 27946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:33857
Jun 20 06:13:57 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
I20260620 06:13:58.072371 28279 heartbeater.cc:464] Registering TS with master...
I20260620 06:13:58.072633 28279 heartbeater.cc:511] Master 127.14.209.190:35759 requested a full tablet report, sending...
I20260620 06:13:58.073273 27946 ts_manager.cc:194] Registered new tserver with Master: 2e6544f55dd24bc69e84ecdf9fe91f52 (127.14.209.130:36769)
I20260620 06:13:58.074010 27946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:37893
I20260620 06:13:58.082268 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:13:58.096179 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.083854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.14.209.190:35759)
0620 06:13:58.086206 (+ 2352us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.086211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.087358 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 06:13:58.087488 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.087491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.087539 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.087632 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:13:58.088476 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.089365 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.090528 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.090879 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.090981 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.092606 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:13:58.092610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:13:58.092613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:13:58.092639 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:13:58.094406 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:58.094880 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:58.094883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:58.094885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:58.094972 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:13:58.095287 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:13:58.095290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:13:58.095292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:13:58.095501 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:13:58.095632 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.095890 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.096010 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2275,"thread_start_us":96,"threads_started":1}
I20260620 06:13:58.100325 27946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:13:58.107363 27946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:13:58.129240 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.117864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:41737 (local address 127.0.0.1:36604)
0620 06:13:58.120570 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:58.120620 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:58.120743 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:58.123831 (+ 3088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:58.123834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:58.123850 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:58.123934 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.123939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.126129 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.126132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.127373 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.127385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.128657 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.128660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.128756 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.128810 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 06:13:58.128923 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:13:58.129066 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2519,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260620 06:13:58.129660 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.118229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.14.209.131:41737)
0620 06:13:58.123245 (+ 5016us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.123252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.123275 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:13:58.123355 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.123360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.123551 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.123692 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:58.124784 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.125955 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.127532 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.128344 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.128472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.128592 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.129425 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.129527 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4908,"thread_start_us":103,"threads_started":1}
I20260620 06:13:58.130956 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.119043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45447 (local address 127.0.0.1:43070)
0620 06:13:58.120614 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:58.120628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:58.120770 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:58.121419 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:58.121425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:58.121446 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:58.121569 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.121575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.123040 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.123043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.124538 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.124551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.127266 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.127269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.130642 (+ 3373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.130666 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:13:58.130759 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:13:58.130828 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"mutex_wait_us":3231,"thread_start_us":76,"threads_started":1}
I20260620 06:13:58.131369 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.119578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.14.209.129:45447)
0620 06:13:58.120487 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.120493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.120705 (+ 212us) server_negotiation.cc:407] Connection header received
0620 06:13:58.120909 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.120914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.121129 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.121241 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:58.121705 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.122916 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.126116 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.126945 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.127075 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.127202 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.130811 (+ 3609us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.130912 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":124,"threads_started":1}
I20260620 06:13:58.132800 28346 tablet_service.cc:1467] Processing CreateTablet for tablet 492a1fc326a746cba422491a7d2f4c56 (DEFAULT_TABLE table=test-table [id=60edbadbc2e14ca39867c209cb2c4cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:13:58.133579 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.119359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:36769 (local address 127.0.0.1:41388)
0620 06:13:58.124269 (+ 4910us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:58.124309 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:58.124533 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:58.125225 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:58.125228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:58.125247 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:58.125330 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.125335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.128691 (+ 3356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.128694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.131802 (+ 3108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.131813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.133204 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.133208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.133309 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.133330 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:13:58.133382 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 06:13:58.133455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4709,"thread_start_us":110,"threads_started":1}
I20260620 06:13:58.133922 28346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492a1fc326a746cba422491a7d2f4c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:58.134011 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.119942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.14.209.130:36769)
0620 06:13:58.120333 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.120340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.124468 (+ 4128us) server_negotiation.cc:407] Connection header received
0620 06:13:58.124739 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.124744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.124927 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.125059 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:58.127406 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.128548 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.131952 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.132784 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.132969 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.133103 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.133761 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.133859 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":83,"threads_started":1}
I20260620 06:13:58.133908 28059 tablet_service.cc:1467] Processing CreateTablet for tablet 492a1fc326a746cba422491a7d2f4c56 (DEFAULT_TABLE table=test-table [id=60edbadbc2e14ca39867c209cb2c4cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:13:58.134819 28059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492a1fc326a746cba422491a7d2f4c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:58.138020 28196 tablet_service.cc:1467] Processing CreateTablet for tablet 492a1fc326a746cba422491a7d2f4c56 (DEFAULT_TABLE table=test-table [id=60edbadbc2e14ca39867c209cb2c4cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:13:58.138912 28196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492a1fc326a746cba422491a7d2f4c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:13:58.142637 28445 tablet_bootstrap.cc:492] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Bootstrap starting.
I20260620 06:13:58.144857 28445 tablet_bootstrap.cc:654] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:58.145701 28445 log.cc:824] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:58.151472 28447 tablet_bootstrap.cc:492] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Bootstrap starting.
I20260620 06:13:58.151667 28448 tablet_bootstrap.cc:492] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: Bootstrap starting.
I20260620 06:13:58.153723 28448 tablet_bootstrap.cc:654] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:58.153723 28447 tablet_bootstrap.cc:654] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Neither blocks nor log segments found. Creating new log.
I20260620 06:13:58.154500 28447 log.cc:824] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:58.156158 28445 tablet_bootstrap.cc:492] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: No bootstrap required, opened a new log
I20260620 06:13:58.156344 28445 ts_tablet_manager.cc:1397] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260620 06:13:58.157593 28447 tablet_bootstrap.cc:492] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: No bootstrap required, opened a new log
I20260620 06:13:58.157799 28447 ts_tablet_manager.cc:1397] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 06:13:58.160766 28445 raft_consensus.cc:348] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.161020 28445 raft_consensus.cc:374] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:58.161087 28445 raft_consensus.cc:729] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9e523a71c84da1bf831fdf65a26244, State: Initialized, Role: FOLLOWER
I20260620 06:13:58.161428 28447 raft_consensus.cc:348] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.161567 28445 consensus_queue.cc:260] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.161649 28447 raft_consensus.cc:374] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:58.161696 28447 raft_consensus.cc:729] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6544f55dd24bc69e84ecdf9fe91f52, State: Initialized, Role: FOLLOWER
I20260620 06:13:58.162282 28447 consensus_queue.cc:260] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.162549 28445 ts_tablet_manager.cc:1428] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:13:58.162882 28412 heartbeater.cc:503] Master 127.14.209.190:35759 was elected leader, sending a full tablet report...
I20260620 06:13:58.162985 28448 log.cc:824] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:13:58.165341 28448 tablet_bootstrap.cc:492] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: No bootstrap required, opened a new log
I20260620 06:13:58.165539 28448 ts_tablet_manager.cc:1397] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260620 06:13:58.169386 28447 ts_tablet_manager.cc:1428] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260620 06:13:58.169734 28279 heartbeater.cc:503] Master 127.14.209.190:35759 was elected leader, sending a full tablet report...
I20260620 06:13:58.169737 28448 raft_consensus.cc:348] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.169986 28448 raft_consensus.cc:374] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:13:58.170043 28448 raft_consensus.cc:729] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38160083e02147af88f9cf8e3bb0b90e, State: Initialized, Role: FOLLOWER
I20260620 06:13:58.170503 28448 consensus_queue.cc:260] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.171483 28448 ts_tablet_manager.cc:1428] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 06:13:58.171782 28144 heartbeater.cc:503] Master 127.14.209.190:35759 was elected leader, sending a full tablet report...
Jun 20 06:13:58 dist-test-slave-zt01 krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for HTTP/127.14.209.148@KRBTEST.COM
W20260620 06:13:58.290019 28413 tablet.cc:2369] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:13:58.299309 28145 tablet.cc:2369] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:13:58.300616 28280 tablet.cc:2369] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:13:58.333413 28450 raft_consensus.cc:482] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:13:58.333638 28450 raft_consensus.cc:504] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.335072 28450 leader_election.cc:288] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38160083e02147af88f9cf8e3bb0b90e (127.14.209.129:45447), 2e6544f55dd24bc69e84ecdf9fe91f52 (127.14.209.130:36769)
I20260620 06:13:58.342001 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.336214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:58351 (local address 127.14.209.129:45447)
0620 06:13:58.336333 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.336338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.337168 (+ 830us) server_negotiation.cc:407] Connection header received
0620 06:13:58.337947 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.337966 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.338065 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.338139 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:58.338506 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.339308 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.340789 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.341341 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.341430 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.341478 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.341787 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.341874 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:13:58.342391 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.335496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45447 (local address 127.14.209.131:58351)
0620 06:13:58.337089 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:58.337103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:58.337682 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:58.338276 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:58.338279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:58.338297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:58.338394 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.338400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.339452 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.339455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.340649 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.340660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.341566 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.341569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.341656 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.341675 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:13:58.342185 (+ 510us) client_negotiation.cc:240] Negotiation successful
0620 06:13:58.342258 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"thread_start_us":107,"threads_started":1}
I20260620 06:13:58.343175 28095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a1fc326a746cba422491a7d2f4c56" candidate_uuid: "fa9e523a71c84da1bf831fdf65a26244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38160083e02147af88f9cf8e3bb0b90e" is_pre_election: true
I20260620 06:13:58.343519 28095 raft_consensus.cc:2454] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa9e523a71c84da1bf831fdf65a26244 in term 0.
I20260620 06:13:58.344030 28302 leader_election.cc:302] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38160083e02147af88f9cf8e3bb0b90e, fa9e523a71c84da1bf831fdf65a26244; no voters:
I20260620 06:13:58.344322 28450 raft_consensus.cc:2781] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:13:58.344391 28450 raft_consensus.cc:482] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:13:58.344429 28450 raft_consensus.cc:3037] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:58.345661 28450 raft_consensus.cc:504] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.345986 28450 leader_election.cc:288] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [CANDIDATE]: Term 1 election: Requested vote from peers 38160083e02147af88f9cf8e3bb0b90e (127.14.209.129:45447), 2e6544f55dd24bc69e84ecdf9fe91f52 (127.14.209.130:36769)
I20260620 06:13:58.346546 28095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a1fc326a746cba422491a7d2f4c56" candidate_uuid: "fa9e523a71c84da1bf831fdf65a26244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38160083e02147af88f9cf8e3bb0b90e"
I20260620 06:13:58.346657 28095 raft_consensus.cc:3037] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:58.347873 28095 raft_consensus.cc:2454] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa9e523a71c84da1bf831fdf65a26244 in term 1.
I20260620 06:13:58.348243 28302 leader_election.cc:302] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38160083e02147af88f9cf8e3bb0b90e, fa9e523a71c84da1bf831fdf65a26244; no voters:
I20260620 06:13:58.348466 28450 raft_consensus.cc:2781] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:13:58.348673 28450 raft_consensus.cc:686] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 1 LEADER]: Becoming Leader. State: Replica: fa9e523a71c84da1bf831fdf65a26244, State: Running, Role: LEADER
I20260620 06:13:58.349015 28450 consensus_queue.cc:237] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } }
I20260620 06:13:58.354801 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.336047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:59409 (local address 127.14.209.130:36769)
0620 06:13:58.336201 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.336206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.342454 (+ 6248us) server_negotiation.cc:407] Connection header received
0620 06:13:58.342655 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.342659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.342708 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.342759 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:13:58.344202 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.344915 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.353448 (+ 8533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.354079 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.354176 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.354226 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.354593 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.354680 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:13:58.355240 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.335914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:36769 (local address 127.14.209.131:59409)
0620 06:13:58.342358 (+ 6444us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:58.342374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:13:58.342538 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:13:58.343918 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:13:58.343921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:13:58.343940 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:13:58.344062 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.344069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.345077 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.345080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.346224 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:13:58.346235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.354342 (+ 8107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.354345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:13:58.354447 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.354469 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:13:58.354991 (+ 522us) client_negotiation.cc:240] Negotiation successful
0620 06:13:58.355101 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6373,"thread_start_us":86,"threads_started":1}
I20260620 06:13:58.355855 27946 catalog_manager.cc:5495] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa9e523a71c84da1bf831fdf65a26244 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "fa9e523a71c84da1bf831fdf65a26244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9e523a71c84da1bf831fdf65a26244" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 41737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 } health_report { overall_health: UNKNOWN } } }
I20260620 06:13:58.358016 28233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a1fc326a746cba422491a7d2f4c56" candidate_uuid: "fa9e523a71c84da1bf831fdf65a26244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" is_pre_election: true
I20260620 06:13:58.358114 28232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492a1fc326a746cba422491a7d2f4c56" candidate_uuid: "fa9e523a71c84da1bf831fdf65a26244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52"
I20260620 06:13:58.358292 28233 raft_consensus.cc:2454] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa9e523a71c84da1bf831fdf65a26244 in term 0.
I20260620 06:13:58.372757 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.367684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.14.209.131:41737)
0620 06:13:58.368029 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 06:13:58.368032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:13:58.368041 (+ 9us) server_negotiation.cc:407] Connection header received
0620 06:13:58.368091 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:13:58.368093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:13:58.368136 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:13:58.368243 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:13:58.368486 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.369235 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.370467 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:13:58.370821 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:13:58.370923 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:13:58.371588 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:13:58.372064 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:13:58.372178 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:13:58.372565 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 06:13:58.372619 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260620 06:13:58.382670 28232 raft_consensus.cc:3037] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:13:58.382898 28095 raft_consensus.cc:1261] T 492a1fc326a746cba422491a7d2f4c56 P 38160083e02147af88f9cf8e3bb0b90e [term 1 FOLLOWER]: Refusing update from remote peer fa9e523a71c84da1bf831fdf65a26244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:13:58.383616 28458 consensus_queue.cc:1035] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38160083e02147af88f9cf8e3bb0b90e" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:13:58.384089 28232 raft_consensus.cc:1261] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 1 FOLLOWER]: Refusing update from remote peer fa9e523a71c84da1bf831fdf65a26244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:13:58.384729 28458 consensus_queue.cc:1035] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6544f55dd24bc69e84ecdf9fe91f52" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:13:58.404349 28461 mvcc.cc:205] Tried to move back new op lower bound from 7298810013207986176 to 7298810013081022464. Current Snapshot: MvccSnapshot[applied={T|T < 7298810013207986176}]
I20260620 06:13:58.421384 28462 mvcc.cc:205] Tried to move back new op lower bound from 7298810013207986176 to 7298810013081022464. Current Snapshot: MvccSnapshot[applied={T|T < 7298810013207986176}]
I20260620 06:13:58.424134 27946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:13:58.424299 27946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:13:58.426816 27946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd9905f24bc44f60beac694c7af57114: Sending DeleteTablet for 3 replicas of tablet 492a1fc326a746cba422491a7d2f4c56
I20260620 06:13:58.427822 28346 tablet_service.cc:1514] Processing DeleteTablet for tablet 492a1fc326a746cba422491a7d2f4c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:13:58 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:36604
I20260620 06:13:58.428251 28059 tablet_service.cc:1514] Processing DeleteTablet for tablet 492a1fc326a746cba422491a7d2f4c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:13:58 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:43070
I20260620 06:13:58.429049 28196 tablet_service.cc:1514] Processing DeleteTablet for tablet 492a1fc326a746cba422491a7d2f4c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:13:58 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:41388
I20260620 06:13:58.430119 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 27984
I20260620 06:13:58.433951 28472 tablet_replica.cc:330] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: stopping tablet replica
I20260620 06:13:58.434305 28472 raft_consensus.cc:2229] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:13:58.434650 28472 raft_consensus.cc:2258] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:13:58.434873 28470 tablet_replica.cc:330] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: stopping tablet replica
I20260620 06:13:58.440304 28470 raft_consensus.cc:2229] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:13:58.441810 28472 ts_tablet_manager.cc:1905] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:13:58.442291 28470 raft_consensus.cc:2258] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:13:58.445188 28472 ts_tablet_manager.cc:1918] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:13:58.446321 28470 ts_tablet_manager.cc:1905] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:13:58.448845 28470 ts_tablet_manager.cc:1918] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:13:58.445425 28472 log.cc:1196] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/492a1fc326a746cba422491a7d2f4c56
I20260620 06:13:58.449344 28472 ts_tablet_manager.cc:1939] T 492a1fc326a746cba422491a7d2f4c56 P 2e6544f55dd24bc69e84ecdf9fe91f52: Deleting consensus metadata
I20260620 06:13:58.449492 28470 log.cc:1196] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/492a1fc326a746cba422491a7d2f4c56
I20260620 06:13:58.451104 27932 catalog_manager.cc:4841] TS 2e6544f55dd24bc69e84ecdf9fe91f52 (127.14.209.130:36769): tablet 492a1fc326a746cba422491a7d2f4c56 (table test-table [id=60edbadbc2e14ca39867c209cb2c4cb6]) successfully deleted
I20260620 06:13:58.451833 28470 ts_tablet_manager.cc:1939] T 492a1fc326a746cba422491a7d2f4c56 P fa9e523a71c84da1bf831fdf65a26244: Deleting consensus metadata
W20260620 06:13:58.452795 27933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:45447 (error 108)
I20260620 06:13:58.453788 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28148
I20260620 06:13:58.457887 27933 catalog_manager.cc:4841] TS fa9e523a71c84da1bf831fdf65a26244 (127.14.209.131:41737): tablet 492a1fc326a746cba422491a7d2f4c56 (table test-table [id=60edbadbc2e14ca39867c209cb2c4cb6]) successfully deleted
I20260620 06:13:58.461934 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:13:58.461584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45447 (local address 127.0.0.1:43072)
0620 06:13:58.461747 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 06:13:58.461835 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 06:13:58.462208 27933 catalog_manager.cc:4584] TS 38160083e02147af88f9cf8e3bb0b90e (127.14.209.129:45447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 492a1fc326a746cba422491a7d2f4c56: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45447: connect: Connection refused (error 111)
I20260620 06:13:58.465716 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28282
I20260620 06:13:58.473038 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 27913
I20260620 06:13:58.479588 15174 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:13:58.823828 15174 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:13:58.823936 15174 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:13:59.167912 15174 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:13:59.168 UTC [26980] LOG: received smart shutdown request
2026-06-20 06:13:59.171 UTC [26980] LOG: background worker "logical replication launcher" (PID 26989) exited with exit code 1
2026-06-20 06:13:59.172 UTC [26984] LOG: shutting down
2026-06-20 06:13:59.337 UTC [26980] LOG: database system is shut down
2026-06-20T06:13:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47567 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:13:59 dist-test-slave-zt01 krb5kdc[28482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:13:59 dist-test-slave-zt01 krb5kdc[28482](info): set up 2 sockets
Jun 20 06:13:59 dist-test-slave-zt01 krb5kdc[28482](info): commencing operation
krb5kdc: starting...
W20260620 06:14:00.483088 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:14:00 dist-test-slave-zt01 krb5kdc[28482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:14:00.597007 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:37707
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:37707,127.14.209.189:42523,127.14.209.188:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:00.699985 28495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:00.700208 28495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:00.700245 28495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:00.704128 28495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:00.704172 28495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:00.704192 28495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:00.704208 28495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:00.704223 28495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:00.709162 28495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:37707,127.14.209.189:42523,127.14.209.188:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:00.710063 28495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:00.710968 28495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:00.718781 28500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:00.718879 28501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:00.718845 28503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:00.718829 28495 server_base.cc:1034] running on GCE node
I20260620 06:14:00.719568 28495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:00.720543 28495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:00.721714 28495 hybrid_clock.cc:648] HybridClock initialized: now 1781936040721692 us; error 35 us; skew 500 ppm
Jun 20 06:14:00 dist-test-slave-zt01 krb5kdc[28482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:00.724248 28495 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:00.725008 28495 webserver.cc:460] Webserver started at http://127.14.209.190:35869/ using document root <none> and password file <none>
I20260620 06:14:00.725512 28495 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:00.725593 28495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:00.725798 28495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:00.727696 28495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "a86163034074473cab5979c0f1294ea4"
format_stamp: "Formatted at 2026-06-20 06:14:00 on dist-test-slave-zt01"
server_key: "bf3019a3219df04e591e23c4f1f382b5"
server_key_iv: "8b9ecc266ba032b3506d5845bbb9c2a1"
server_key_version: "encryptionkey@0"
I20260620 06:14:00.728223 28495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "a86163034074473cab5979c0f1294ea4"
format_stamp: "Formatted at 2026-06-20 06:14:00 on dist-test-slave-zt01"
server_key: "bf3019a3219df04e591e23c4f1f382b5"
server_key_iv: "8b9ecc266ba032b3506d5845bbb9c2a1"
server_key_version: "encryptionkey@0"
I20260620 06:14:00.731540 28495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:14:00.733721 28510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:00.734799 28495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:00.734931 28495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "a86163034074473cab5979c0f1294ea4"
format_stamp: "Formatted at 2026-06-20 06:14:00 on dist-test-slave-zt01"
server_key: "bf3019a3219df04e591e23c4f1f382b5"
server_key_iv: "8b9ecc266ba032b3506d5845bbb9c2a1"
server_key_version: "encryptionkey@0"
I20260620 06:14:00.735030 28495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:00.756589 28495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:00.759267 28495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:00.759470 28495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:00.765820 28495 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:37707
I20260620 06:14:00.766077 28562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:37707 every 8 connection(s)
I20260620 06:14:00.766701 28495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:00.769164 28563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:00.772035 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28495
I20260620 06:14:00.772172 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:14:00.772449 15174 external_mini_cluster.cc:1428] Setting key 951a33890bb7da64733409eedbd9a89f
I20260620 06:14:00.772806 28563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.209.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.189@KRBTEST.COM" created.
Jun 20 06:14:00 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:00.788858 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.775253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:37707 (local address 127.0.0.1:44324)
0620 06:14:00.775750 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.775801 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:00.776970 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:00.777649 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:00.777657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:00.778141 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:00.778524 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:00.778543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.779747 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.779751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:00.780980 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:00.780995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.781567 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.781570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:00.781657 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:00.782192 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:00.782208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:00.784159 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:00.786289 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:00.786296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:00.786302 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:00.786513 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:00.786874 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:00.786876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:00.786878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:00.786974 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:00.787682 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:00.787687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:00.787796 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 06:14:00.787928 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 06:14:00.788156 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
I20260620 06:14:00.788864 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.775253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.14.209.190:37707)
0620 06:14:00.775751 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:00.775768 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:00.775907 (+ 139us) server_negotiation.cc:407] Connection header received
0620 06:14:00.777278 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:00.777282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:00.777446 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:00.777590 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:00.778682 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.779620 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.781105 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.781446 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.781551 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:00.784296 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:00.784306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:00.784320 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:00.784352 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:00.786187 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:00.786621 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:00.786630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:00.786634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:00.786766 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:00.787083 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:00.787090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:00.787094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:00.787572 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:00.787710 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:00.788039 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 06:14:00.788261 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":393,"spinlock_wait_cycles":27392,"thread_start_us":157,"threads_started":1}
I20260620 06:14:00.791145 28563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:14:00.791785 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.791566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60610)
0620 06:14:00.791670 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.791739 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 06:14:00.791989 28514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
I20260620 06:14:00.792747 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.792569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60626)
0620 06:14:00.792660 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.792702 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:14:00.792917 28563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:42523: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab.
Entry for principal kudu/127.14.209.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab.
I20260620 06:14:00.800148 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:42523
--webserver_interface=127.14.209.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab
--principal=kudu/127.14.209.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:37707,127.14.209.189:42523,127.14.209.188:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:00.842156 28563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } attempt: 1
I20260620 06:14:00.843120 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.842683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60628)
0620 06:14:00.842869 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.842970 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 06:14:00.843972 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.843725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60632)
0620 06:14:00.843840 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.843892 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:14:00.844131 28563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:42523: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
W20260620 06:14:00.907881 28570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:00.908092 28570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:00.908128 28570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:00.912079 28570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:00.912123 28570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:00.912143 28570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:00.912159 28570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:00.912207 28570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:00.912403 28563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } attempt: 2
I20260620 06:14:00.913216 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.912884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60634)
0620 06:14:00.913056 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.913129 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 06:14:00.913975 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.913754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60650)
0620 06:14:00.913854 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.913904 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:14:00.914104 28563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:42523: Network error: Client connection negotiation failed: client connection to 127.14.209.189:42523: connect: Connection refused (error 111)
I20260620 06:14:00.917686 28570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:37707,127.14.209.189:42523,127.14.209.188:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab
--principal=kudu/127.14.209.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.209.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:00.918569 28570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:00.919512 28570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:00.927203 28578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:00.927212 28576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:00.927502 28575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:00.927541 28570 server_base.cc:1034] running on GCE node
I20260620 06:14:00.928082 28570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:00.929055 28570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:00.930214 28570 hybrid_clock.cc:648] HybridClock initialized: now 1781936040930194 us; error 31 us; skew 500 ppm
Jun 20 06:14:00 dist-test-slave-zt01 krb5kdc[28482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:00.932866 28570 init.cc:377] Logged in from keytab as kudu/127.14.209.189@KRBTEST.COM (short username kudu)
I20260620 06:14:00.933617 28570 webserver.cc:460] Webserver started at http://127.14.209.189:38211/ using document root <none> and password file <none>
I20260620 06:14:00.934099 28570 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:00.934157 28570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:00.934306 28570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:00.936213 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data/instance:
uuid: "b8c59d992d4d41d7bf3ba0995407d3c6"
format_stamp: "Formatted at 2026-06-20 06:14:00 on dist-test-slave-zt01"
server_key: "03246b6ff2946ee093177bad1e964cdb"
server_key_iv: "b12067105088c64ee1c50ae43baf5668"
server_key_version: "encryptionkey@0"
I20260620 06:14:00.936682 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal/instance:
uuid: "b8c59d992d4d41d7bf3ba0995407d3c6"
format_stamp: "Formatted at 2026-06-20 06:14:00 on dist-test-slave-zt01"
server_key: "03246b6ff2946ee093177bad1e964cdb"
server_key_iv: "b12067105088c64ee1c50ae43baf5668"
server_key_version: "encryptionkey@0"
I20260620 06:14:00.940245 28570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:00.942350 28585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:00.943542 28570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:00.943642 28570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal
uuid: "b8c59d992d4d41d7bf3ba0995407d3c6"
format_stamp: "Formatted at 2026-06-20 06:14:00 on dist-test-slave-zt01"
server_key: "03246b6ff2946ee093177bad1e964cdb"
server_key_iv: "b12067105088c64ee1c50ae43baf5668"
server_key_version: "encryptionkey@0"
I20260620 06:14:00.943720 28570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:00.967290 28570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:00.970142 28570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:00.970288 28570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:00.976964 28570 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.189:42523
I20260620 06:14:00.977233 28637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.189:42523 every 8 connection(s)
I20260620 06:14:00.977846 28570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
I20260620 06:14:00.980352 28638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:00.983835 28638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:14:00.986649 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28570
I20260620 06:14:00.986785 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-1/wal/instance
I20260620 06:14:00.987108 15174 external_mini_cluster.cc:1428] Setting key 290e4145d8be44cab93d518734bc66f1
Jun 20 06:14:00 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.188@KRBTEST.COM; defaulting to no policy
I20260620 06:14:00.996798 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.985493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.14.209.190:37707)
0620 06:14:00.985628 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:00.985632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:00.986311 (+ 679us) server_negotiation.cc:407] Connection header received
0620 06:14:00.987464 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:00.987467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:00.987516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:00.987587 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:00.989064 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.989584 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.990688 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.990925 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.991043 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:00.993325 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:00.993328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:00.993330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:00.993354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:00.994776 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:00.995331 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:00.995335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:00.995337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:00.995417 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:00.995757 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:00.995760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:00.995761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:00.995922 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:00.995995 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:00.996584 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 06:14:00.996682 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.188@KRBTEST.COM" created.
I20260620 06:14:00.997660 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.985796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:37707 (local address 127.0.0.1:44340)
0620 06:14:00.986176 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.986213 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:00.987240 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:00.987771 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:00.987783 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:00.988249 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:00.988919 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:00.988934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.989709 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.989712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:00.990562 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:00.990571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:00.991095 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:00.991100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:00.991267 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:00.991844 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:00.991859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:00.993166 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:00.994883 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:00.994889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:00.994901 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:00.995213 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:00.995538 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:00.995541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:00.995542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:00.995668 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:00.996060 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:00.996065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:00.996325 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 06:14:00.996503 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 06:14:00.996720 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260620 06:14:00.999025 28638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:14:00 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
I20260620 06:14:01.005272 28563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } attempt: 3
I20260620 06:14:01.007829 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.999554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60662)
0620 06:14:00.999660 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:00.999671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:00.999729 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.000294 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.000296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.000457 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.000567 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.000576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.001489 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.001492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.001998 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.002005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.002554 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.002558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.002646 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.003135 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.003146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.004233 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.006113 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.006119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.006124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.006452 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.006783 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.006785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.006787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.006856 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.007326 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.007328 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.007401 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.007483 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.007598 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":2}
I20260620 06:14:01.007900 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:00.999698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.14.209.189:42523)
0620 06:14:00.999991 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:00.999999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.000023 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:14:01.000054 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.000057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.000182 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.000317 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.000667 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.001373 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.002103 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.002444 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.002560 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.004364 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.004371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.004381 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.004399 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.005984 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.006569 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.006578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.006582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.006678 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.006934 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.006937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.006939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.007235 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.007376 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.007585 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.007735 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":146,"threads_started":1}
I20260620 06:14:01.010043 28638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:14:01.010811 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.010520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39340)
0620 06:14:01.010667 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.010749 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 06:14:01.010993 28588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
I20260620 06:14:01.011821 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.011597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39342)
0620 06:14:01.011726 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.011775 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 06:14:01.011999 28638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:37277: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab.
Entry for principal kudu/127.14.209.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab.
I20260620 06:14:01.014668 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:37277
--webserver_interface=127.14.209.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab
--principal=kudu/127.14.209.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:37707,127.14.209.189:42523,127.14.209.188:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:01.015043 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.005777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60664)
0620 06:14:01.005977 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.005996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:01.006083 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.006365 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.006368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.006543 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.006644 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.006649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.007683 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.007686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.009079 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.009090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.009584 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.009588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.009700 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.010130 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.010143 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.011591 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.013312 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.013314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.013316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.013499 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.013938 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.013940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.013941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.014017 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.014394 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.014398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.014534 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.014655 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.014796 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:14:01.015223 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.005843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.14.209.189:42523)
0620 06:14:01.006182 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.006189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.006198 (+ 9us) server_negotiation.cc:407] Connection header received
0620 06:14:01.006223 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.006225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.006254 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.006318 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.006749 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.007569 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.009200 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.009473 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.009582 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.011727 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.011730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.011733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.011754 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.013211 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.013708 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.013712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.013714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.013803 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.014119 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.014121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.014122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.014270 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.014378 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.014828 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.014957 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
I20260620 06:14:01.015851 28563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:14:01.016464 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.016251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39352)
0620 06:14:01.016353 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.016401 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 06:14:01.017102 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.016886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39366)
0620 06:14:01.017010 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.017054 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 06:14:01.017211 28563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:37277: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
I20260620 06:14:01.048357 28563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } attempt: 1
I20260620 06:14:01.048969 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.048739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39374)
0620 06:14:01.048856 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.048911 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 06:14:01.049580 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.049387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39388)
0620 06:14:01.049481 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.049536 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 06:14:01.049688 28563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:37277: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
I20260620 06:14:01.061234 28638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } attempt: 1
I20260620 06:14:01.061849 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.061608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39392)
0620 06:14:01.061728 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.061781 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 06:14:01.062462 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.062262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39400)
0620 06:14:01.062357 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.062405 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:14:01.062579 28638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:37277: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
W20260620 06:14:01.119561 28646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:01.119819 28646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:01.119886 28646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:01.124033 28646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:01.124100 28646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:01.124157 28646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:01.124207 28646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:01.124243 28646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:01.124845 28563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } attempt: 2
I20260620 06:14:01.125658 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.125333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39410)
0620 06:14:01.125501 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.125580 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 06:14:01.126317 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.126121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39412)
0620 06:14:01.126221 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.126268 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:14:01.126434 28563 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:37277: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
I20260620 06:14:01.129531 28646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:37707,127.14.209.189:42523,127.14.209.188:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab
--principal=kudu/127.14.209.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.209.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:01.130481 28646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:01.130717 28638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } attempt: 2
I20260620 06:14:01.131421 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.131146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39424)
0620 06:14:01.131287 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.131339 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 06:14:01.131611 28646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:14:01.132090 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.131899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39436)
0620 06:14:01.131987 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.132040 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 06:14:01.132203 28638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:37277: Network error: Client connection negotiation failed: client connection to 127.14.209.188:37277: connect: Connection refused (error 111)
W20260620 06:14:01.139592 28654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:01.139622 28651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:01.139642 28652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:01.139811 28646 server_base.cc:1034] running on GCE node
I20260620 06:14:01.140329 28646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:01.141310 28646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:01.142743 28646 hybrid_clock.cc:648] HybridClock initialized: now 1781936041142701 us; error 67 us; skew 500 ppm
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:01.145546 28646 init.cc:377] Logged in from keytab as kudu/127.14.209.188@KRBTEST.COM (short username kudu)
I20260620 06:14:01.146384 28646 webserver.cc:460] Webserver started at http://127.14.209.188:35419/ using document root <none> and password file <none>
I20260620 06:14:01.146898 28646 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:01.146986 28646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:01.147243 28646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:01.149134 28646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data/instance:
uuid: "e6089dc389214c44a65d5cb55920dc7a"
format_stamp: "Formatted at 2026-06-20 06:14:01 on dist-test-slave-zt01"
server_key: "fd247925ea6a10d74f16210459325601"
server_key_iv: "0d83455ef046ea60a1c67833d418ac77"
server_key_version: "encryptionkey@0"
I20260620 06:14:01.149689 28646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal/instance:
uuid: "e6089dc389214c44a65d5cb55920dc7a"
format_stamp: "Formatted at 2026-06-20 06:14:01 on dist-test-slave-zt01"
server_key: "fd247925ea6a10d74f16210459325601"
server_key_iv: "0d83455ef046ea60a1c67833d418ac77"
server_key_version: "encryptionkey@0"
I20260620 06:14:01.153499 28646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:14:01.155953 28661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:01.157172 28646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:01.157303 28646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal
uuid: "e6089dc389214c44a65d5cb55920dc7a"
format_stamp: "Formatted at 2026-06-20 06:14:01 on dist-test-slave-zt01"
server_key: "fd247925ea6a10d74f16210459325601"
server_key_iv: "0d83455ef046ea60a1c67833d418ac77"
server_key_version: "encryptionkey@0"
I20260620 06:14:01.157405 28646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:01.166128 28646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:01.168838 28646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:01.169018 28646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:01.175702 28646 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.188:37277
I20260620 06:14:01.175998 28713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.188:37277 every 8 connection(s)
I20260620 06:14:01.176587 28646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
I20260620 06:14:01.179186 28714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:01.180267 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28646
I20260620 06:14:01.180405 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935885709390-15174-0/minicluster-data/master-2/wal/instance
I20260620 06:14:01.180676 15174 external_mini_cluster.cc:1428] Setting key d70e530fc0403afd653c0b2e73187c2b
I20260620 06:14:01.183108 28714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:01.194679 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.184865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.14.209.190:37707)
0620 06:14:01.185006 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.185010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.185282 (+ 272us) server_negotiation.cc:407] Connection header received
0620 06:14:01.185407 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.185410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.185451 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.185516 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.186101 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.186652 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.187617 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.188000 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.188124 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.190996 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.191006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.191010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.191036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.192827 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.193332 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.193335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.193337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.193413 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.193694 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.193697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.193699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.193898 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.193962 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.194400 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.194525 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:14:01.195680 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.184802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:37707 (local address 127.0.0.1:44346)
0620 06:14:01.185163 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.185205 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:01.185313 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.185579 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.185586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.185877 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.185986 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.185991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.186761 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.186764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.187516 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.187523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.188136 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.188138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.188214 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.188699 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.188722 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.190536 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.192947 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.192955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.192961 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.193224 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.193520 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.193522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.193524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.193602 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.194042 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.194048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.194178 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.194806 (+ 628us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.194976 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260620 06:14:01.198776 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.181825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.14.209.188:37277)
0620 06:14:01.182180 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.182194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.182225 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:14:01.182822 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.182837 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.183192 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.183468 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.184268 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.185473 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.186381 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.186908 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.187075 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.189931 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.189937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.189949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.189974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.191873 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.192646 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.192651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.192656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.192828 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.193153 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.193156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.193158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.193695 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.193912 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.194362 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.194732 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":118,"threads_started":1}
I20260620 06:14:01.200472 28714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
I20260620 06:14:01.209142 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.200914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.14.209.190:37707)
0620 06:14:01.201040 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.201046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.201123 (+ 77us) server_negotiation.cc:407] Connection header received
0620 06:14:01.201232 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.201234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.201265 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.201351 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.201921 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.202376 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.203179 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.203378 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.203483 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.206096 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.206100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.206103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.206126 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.207614 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.208019 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.208023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.208024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.208101 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.208356 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.208360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.208362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.208548 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.208772 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.208880 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.208990 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:14:01.209460 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.200885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:42523 (local address 127.0.0.1:60676)
0620 06:14:01.201295 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.201305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:01.201393 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.201614 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.201617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.201771 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.201851 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.201856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.202563 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.202568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.203103 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.203112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.203561 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.203565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.203659 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.204156 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.204169 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.206301 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.208068 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.208073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.208075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.208263 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.208541 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.208544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.208546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.208628 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.208981 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.208984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.209092 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.209197 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.209315 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352}
I20260620 06:14:01.209496 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.201023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.14.209.189:42523)
0620 06:14:01.201167 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.201171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.201355 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:14:01.201473 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.201476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.201518 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.201602 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.201955 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.202458 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.203213 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.203458 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.203563 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.206416 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.206423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.206429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.206460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.207931 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.208366 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.208369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.208371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.208448 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.208722 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.208725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.208727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.208892 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.208990 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.209246 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.209359 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:14:01.210166 28714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:14:01.219348 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.210233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.14.209.189:42523)
0620 06:14:01.210333 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.210337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.210393 (+ 56us) server_negotiation.cc:407] Connection header received
0620 06:14:01.210517 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.210519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.210550 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.210636 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.211476 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.212180 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.213190 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.213420 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.213518 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.215964 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.215966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.215968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.216011 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.217726 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.218113 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.218117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.218122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.218208 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.218482 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.218486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.218487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.218728 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.218854 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.219084 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.219195 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:14:01.219859 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.210623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39468)
0620 06:14:01.211158 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.211169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:01.211233 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.211446 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.211448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.211560 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.211621 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.211625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.212980 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.212983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.213526 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.213535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.214181 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.214185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.214280 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.214705 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.214714 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.216331 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.218142 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.218144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.218145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.218383 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.218853 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.218856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.218858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.218946 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.219372 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.219375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.219480 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.219586 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.219703 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":1}
I20260620 06:14:01.219880 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.210677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.14.209.188:37277)
0620 06:14:01.210777 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.210780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.211227 (+ 447us) server_negotiation.cc:407] Connection header received
0620 06:14:01.211291 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.211294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.211334 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.211417 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.211718 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.212876 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.213674 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.214051 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.214182 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.216468 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.216472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.216474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.216500 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.218046 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.218483 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.218486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.218488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.218580 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.219037 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.219040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.219042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.219269 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.219368 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.219653 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.219751 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":43}
I20260620 06:14:01.222502 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:14:01.223136 28714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: Bootstrap starting.
I20260620 06:14:01.223338 28638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } attempt: 3
I20260620 06:14:01.225932 28714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:01.226351 28563 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } attempt: 3
I20260620 06:14:01.226781 28714 log.cc:824] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:01.228848 28714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: No bootstrap required, opened a new log
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:14:01.233513 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.223686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39484)
0620 06:14:01.223817 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.223830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:01.223927 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.224213 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.224216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.224400 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.224511 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.224518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.225448 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.225451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.226247 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.226254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.227981 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.227984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.228063 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.228481 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.228494 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.229931 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.231834 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.231837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.231839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.232096 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.232435 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.232438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.232440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.232530 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.233039 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.233042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.233130 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.233233 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.233366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 06:14:01.233601 28714 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.233839 28714 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:01.233902 28714 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6089dc389214c44a65d5cb55920dc7a, State: Initialized, Role: FOLLOWER
I20260620 06:14:01.234038 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.223721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.14.209.188:37277)
0620 06:14:01.224055 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.224058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.224067 (+ 9us) server_negotiation.cc:407] Connection header received
0620 06:14:01.224091 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.224093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.224120 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.224199 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.224637 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.225314 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.227479 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.227798 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.227884 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.230076 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.230081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.230087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.230114 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.231667 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.232241 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.232246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.232251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.232333 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.232632 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.232636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.232640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.232896 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.232994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.233789 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.233900 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260620 06:14:01.234359 28714 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.234828 28729 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.234972 28729 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:14:01.235821 28714 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:01.237406 28638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6: Bootstrap starting.
Jun 20 06:14:01 dist-test-slave-zt01 krb5kdc[28482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:14:01.238174 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.224615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.14.209.190:37707)
0620 06:14:01.224715 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.224719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.224925 (+ 206us) server_negotiation.cc:407] Connection header received
0620 06:14:01.225101 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.225104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.225145 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.225193 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.226109 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.226813 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.227991 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.228584 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.228660 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.231157 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.231161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.231163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.231191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.232823 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.234016 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.234021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.234023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.234105 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.234402 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.234406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.234407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.234602 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.234705 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.237871 (+ 3166us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.237995 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:14:01.240819 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.225283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.14.209.189:42523)
0620 06:14:01.225389 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.225393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.226212 (+ 819us) server_negotiation.cc:407] Connection header received
0620 06:14:01.226372 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.226376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.226416 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.226463 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.227457 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.228131 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.229261 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.229613 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.229691 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.235048 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.235074 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.235077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.235101 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.237197 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.239039 (+ 1842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.239042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.239044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.239158 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.239495 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.239498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.239500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.239687 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.239788 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.240307 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.240413 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 06:14:01.241429 28740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:14:01.241493 28740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:14:01.243644 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.226710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:37277 (local address 127.0.0.1:39504)
0620 06:14:01.226983 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:01.226999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:01.227093 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:01.230550 (+ 3457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:01.230553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:01.230732 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:01.230840 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.230846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.233375 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.233378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.234141 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:01.234153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.235745 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.235748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:01.235823 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.236525 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:01.236539 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:01.237877 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:01.241934 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.241938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.241940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.242182 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.242527 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:01.242531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:01.242532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:01.242641 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:01.243155 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:01.243160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:01.243257 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 06:14:01.243350 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:14:01.243491 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260620 06:14:01.240074 28646 master_runner.cc:186] Error getting master registration for 127.14.209.190:37707: OK, instance_id { permanent_uuid: "a86163034074473cab5979c0f1294ea4" instance_seqno: 1781936040764088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:14:01.244335 28646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:14:01.244470 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.227236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.14.209.188:37277)
0620 06:14:01.228328 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.228336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.228374 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:14:01.228417 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.228420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.230373 (+ 1953us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.230964 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.231053 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.231859 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.235273 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.235617 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.236113 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.238051 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.238054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.238057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.238079 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.241804 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.242300 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.242302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.242304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.242380 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.242751 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.242753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.242755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.242996 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.244004 (+ 1008us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.244076 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.244177 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":101,"threads_started":1}
I20260620 06:14:01.246969 28563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: Bootstrap starting.
I20260620 06:14:01.248955 28563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:01.249588 28638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:01.249630 28563 log.cc:824] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:01.250365 28638 log.cc:824] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:01.251535 28563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: No bootstrap required, opened a new log
I20260620 06:14:01.252213 28638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6: No bootstrap required, opened a new log
I20260620 06:14:01.255400 28563 raft_consensus.cc:348] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.255620 28563 raft_consensus.cc:374] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:01.255681 28563 raft_consensus.cc:729] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86163034074473cab5979c0f1294ea4, State: Initialized, Role: FOLLOWER
I20260620 06:14:01.256102 28638 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.256151 28563 consensus_queue.cc:260] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.256347 28638 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:01.256448 28638 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c59d992d4d41d7bf3ba0995407d3c6, State: Initialized, Role: FOLLOWER
I20260620 06:14:01.256907 28638 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.257403 28563 sys_catalog.cc:564] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:01.258143 28638 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:01.258848 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:01.224945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.14.209.188:37277)
0620 06:14:01.226013 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:01.226021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:01.232150 (+ 6129us) server_negotiation.cc:407] Connection header received
0620 06:14:01.232326 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:01.232329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:01.232370 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:01.232442 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:01.244940 (+ 12498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.251554 (+ 6614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.252795 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:01.253183 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:01.254896 (+ 1713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:01.255016 (+ 120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:01.255019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:01.255022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:01.255039 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:01.256759 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.257243 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.257247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.257249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.257321 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:01.257602 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:01.257605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:01.257607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:01.257791 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:01.258510 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:01.258580 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 06:14:01.258684 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1010,"thread_start_us":89,"threads_started":1}
I20260620 06:14:01.259580 28744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.259703 28744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:14:01.260306 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.260439 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 06:14:01.260775 28516 tablet.cc:2369] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:14:01.263657 28766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:14:01.263731 28766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 06:14:01.264007 28767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:14:01.264150 28767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:14:01.266965 28745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:01.267109 28745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.267882 28745 leader_election.cc:288] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a86163034074473cab5979c0f1294ea4 (127.14.209.190:37707), e6089dc389214c44a65d5cb55920dc7a (127.14.209.188:37277)
I20260620 06:14:01.268090 28495 master_runner.cc:263] No leader master found from master a86163034074473cab5979c0f1294ea4
I20260620 06:14:01.268152 28495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:14:01.268384 28537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a86163034074473cab5979c0f1294ea4" is_pre_election: true
I20260620 06:14:01.268462 28688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6089dc389214c44a65d5cb55920dc7a" is_pre_election: true
I20260620 06:14:01.268623 28537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c59d992d4d41d7bf3ba0995407d3c6 in term 0.
I20260620 06:14:01.268685 28688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c59d992d4d41d7bf3ba0995407d3c6 in term 0.
I20260620 06:14:01.268999 28588 leader_election.cc:302] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a86163034074473cab5979c0f1294ea4, b8c59d992d4d41d7bf3ba0995407d3c6; no voters:
I20260620 06:14:01.269246 28745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:14:01.269321 28745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:14:01.269362 28745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:01.269690 28570 master_runner.cc:263] No leader master found from master b8c59d992d4d41d7bf3ba0995407d3c6
I20260620 06:14:01.269760 28570 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:14:01.270527 28745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.270826 28745 leader_election.cc:288] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [CANDIDATE]: Term 1 election: Requested vote from peers a86163034074473cab5979c0f1294ea4 (127.14.209.190:37707), e6089dc389214c44a65d5cb55920dc7a (127.14.209.188:37277)
I20260620 06:14:01.271234 28688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6089dc389214c44a65d5cb55920dc7a"
I20260620 06:14:01.271349 28688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:01.271474 28537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a86163034074473cab5979c0f1294ea4"
I20260620 06:14:01.271605 28537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:01.272514 28537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c59d992d4d41d7bf3ba0995407d3c6 in term 1.
I20260620 06:14:01.272871 28688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c59d992d4d41d7bf3ba0995407d3c6 in term 1.
I20260620 06:14:01.272923 28588 leader_election.cc:302] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a86163034074473cab5979c0f1294ea4, b8c59d992d4d41d7bf3ba0995407d3c6; no voters:
I20260620 06:14:01.273077 28745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:01.273252 28745 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [term 1 LEADER]: Becoming Leader. State: Replica: b8c59d992d4d41d7bf3ba0995407d3c6, State: Running, Role: LEADER
I20260620 06:14:01.273525 28745 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } }
I20260620 06:14:01.275372 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c59d992d4d41d7bf3ba0995407d3c6. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.275484 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:01.275833 28772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:01.276667 28772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:01.281720 28688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [term 1 FOLLOWER]: Refusing update from remote peer b8c59d992d4d41d7bf3ba0995407d3c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:01.282380 28537 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [term 1 FOLLOWER]: Refusing update from remote peer b8c59d992d4d41d7bf3ba0995407d3c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:01.282356 28770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:01.282920 28770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:01.286361 28729 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c59d992d4d41d7bf3ba0995407d3c6. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.286492 28729 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:14:01.289487 28744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c59d992d4d41d7bf3ba0995407d3c6. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.289604 28744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:14:01.293032 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.293138 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:01.294116 28744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.294222 28744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:14:01.294756 28776 mvcc.cc:205] Tried to move back new op lower bound from 7298810025082097664 to 7298810025058684928. Current Snapshot: MvccSnapshot[applied={T|T < 7298810025082097664}]
I20260620 06:14:01.296550 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.296658 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:01.297608 28729 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86163034074473cab5979c0f1294ea4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 37707 } } peers { permanent_uuid: "b8c59d992d4d41d7bf3ba0995407d3c6" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 42523 } } peers { permanent_uuid: "e6089dc389214c44a65d5cb55920dc7a" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 37277 } } }
I20260620 06:14:01.297711 28729 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:14:01.298138 28774 mvcc.cc:205] Tried to move back new op lower bound from 7298810025082097664 to 7298810025058684928. Current Snapshot: MvccSnapshot[applied={T|T < 7298810025082097664}]
I20260620 06:14:01.299515 28772 catalog_manager.cc:1347] Generated new cluster ID: 4fbe450748e74a6a8d214675039da7f1
I20260620 06:14:01.299580 28772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:14:01.320133 28772 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:14:01.321344 28772 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:14:01.335693 28772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c59d992d4d41d7bf3ba0995407d3c6: Generated new TSK 0
I20260620 06:14:01.336432 28772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:14:02.242735 28740 catalog_manager.cc:1259] Loaded cluster ID: 4fbe450748e74a6a8d214675039da7f1
I20260620 06:14:02.242810 28740 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: loading cluster ID for follower catalog manager: success
I20260620 06:14:02.244414 28740 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: acquiring CA information for follower catalog manager: success
I20260620 06:14:02.245486 28740 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6089dc389214c44a65d5cb55920dc7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:14:02.264717 28766 catalog_manager.cc:1259] Loaded cluster ID: 4fbe450748e74a6a8d214675039da7f1
I20260620 06:14:02.264778 28766 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: loading cluster ID for follower catalog manager: success
I20260620 06:14:02.266309 28766 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: acquiring CA information for follower catalog manager: success
I20260620 06:14:02.267405 28766 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a86163034074473cab5979c0f1294ea4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:14:02.398748 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28495
I20260620 06:14:02.404834 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28570
I20260620 06:14:02.410467 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28646
2026-06-20T06:14:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2959 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:14:02 dist-test-slave-zt01 krb5kdc[28788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:14:02 dist-test-slave-zt01 krb5kdc[28788](info): set up 2 sockets
Jun 20 06:14:02 dist-test-slave-zt01 krb5kdc[28788](info): commencing operation
krb5kdc: starting...
W20260620 06:14:03.443199 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:14:03 dist-test-slave-zt01 krb5kdc[28788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:14:03.562350 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34497
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:03.664642 28801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:03.664866 28801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:03.664904 28801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:03.668713 28801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:03.668759 28801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:03.668778 28801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:03.668795 28801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:03.668812 28801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:03.673686 28801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:34497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:03.674474 28801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:03.675375 28801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:03.683022 28809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:03.683029 28807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:03.683252 28806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:03.683349 28801 server_base.cc:1034] running on GCE node
I20260620 06:14:03.684019 28801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:03.684985 28801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:03.686179 28801 hybrid_clock.cc:648] HybridClock initialized: now 1781936043686166 us; error 25 us; skew 500 ppm
Jun 20 06:14:03 dist-test-slave-zt01 krb5kdc[28788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936043, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:03.688828 28801 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:03.689668 28801 webserver.cc:460] Webserver started at http://127.14.209.190:42109/ using document root <none> and password file <none>
I20260620 06:14:03.690173 28801 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:03.690261 28801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:03.690488 28801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:03.692345 28801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "269ff3d3c1cf4ec8bd2461b535021a75"
format_stamp: "Formatted at 2026-06-20 06:14:03 on dist-test-slave-zt01"
server_key: "e0755f06404503b682eee399ddbfdd75"
server_key_iv: "635ca88daac52330feee84cb64e9be00"
server_key_version: "encryptionkey@0"
I20260620 06:14:03.692878 28801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "269ff3d3c1cf4ec8bd2461b535021a75"
format_stamp: "Formatted at 2026-06-20 06:14:03 on dist-test-slave-zt01"
server_key: "e0755f06404503b682eee399ddbfdd75"
server_key_iv: "635ca88daac52330feee84cb64e9be00"
server_key_version: "encryptionkey@0"
I20260620 06:14:03.696198 28801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:03.698338 28816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:03.699541 28801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:03.699674 28801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "269ff3d3c1cf4ec8bd2461b535021a75"
format_stamp: "Formatted at 2026-06-20 06:14:03 on dist-test-slave-zt01"
server_key: "e0755f06404503b682eee399ddbfdd75"
server_key_iv: "635ca88daac52330feee84cb64e9be00"
server_key_version: "encryptionkey@0"
I20260620 06:14:03.699779 28801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:03.712476 28801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:03.715461 28801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:03.715668 28801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:03.722313 28801 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:34497
I20260620 06:14:03.722337 28868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:34497 every 8 connection(s)
I20260620 06:14:03.723160 28801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:03.725560 28869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:03.727203 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28801
I20260620 06:14:03.727325 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:14:03.727619 15174 external_mini_cluster.cc:1428] Setting key ca5f752c6a6f299ca8c4c9b3f795f75f
I20260620 06:14:03.731453 28869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75: Bootstrap starting.
I20260620 06:14:03.733489 28869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:03.734153 28869 log.cc:824] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:14:03 dist-test-slave-zt01 krb5kdc[28788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:03.736219 28869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75: No bootstrap required, opened a new log
I20260620 06:14:03.739835 28869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34497 } }
I20260620 06:14:03.740056 28869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:03.740149 28869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269ff3d3c1cf4ec8bd2461b535021a75, State: Initialized, Role: FOLLOWER
I20260620 06:14:03.740602 28869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34497 } }
I20260620 06:14:03.740751 28869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:14:03.740837 28869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:14:03.740952 28869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:03.741253 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:03.728849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.14.209.190:34497)
0620 06:14:03.729237 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:03.729250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:03.729282 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:14:03.729916 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:03.729932 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:03.730225 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:03.730535 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:03.731334 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:03.732329 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:03.733184 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:03.733690 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:03.733856 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:03.736740 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:03.736744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:03.736755 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:03.736778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:03.738432 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:03.738913 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:03.738917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:03.738921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:03.739026 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:03.739364 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:03.739367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:03.739369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:03.739708 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:03.739918 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:03.740157 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 06:14:03.740441 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260620 06:14:03.741994 28869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34497 } }
I20260620 06:14:03.742328 28869 leader_election.cc:302] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 269ff3d3c1cf4ec8bd2461b535021a75; no voters:
I20260620 06:14:03.742635 28869 leader_election.cc:288] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:14:03.742717 28874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:03.742883 28874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [term 1 LEADER]: Becoming Leader. State: Replica: 269ff3d3c1cf4ec8bd2461b535021a75, State: Running, Role: LEADER
I20260620 06:14:03.743582 28874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34497 } }
I20260620 06:14:03.743826 28869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:03.744246 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:14:03.745160 28875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34497 } } }
I20260620 06:14:03.745287 28875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:03.745643 28876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 269ff3d3c1cf4ec8bd2461b535021a75. Latest consensus state: current_term: 1 leader_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269ff3d3c1cf4ec8bd2461b535021a75" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34497 } } }
I20260620 06:14:03.745776 28876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75 [sys.catalog]: This master's current role is: LEADER
W20260620 06:14:03.748600 28889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:14:03.748690 28889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:14:03.748766 28885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:03.749454 28885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:03.754091 28885 catalog_manager.cc:1347] Generated new cluster ID: 6ce525f718e2403db9dd3d7a1c88cdb4
I20260620 06:14:03.754168 28885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:14:03.761149 28885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:14:03.761924 28885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:14:03.769129 28885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 269ff3d3c1cf4ec8bd2461b535021a75: Generated new TSK 0
I20260620 06:14:03.769702 28885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:14:03.799867 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28801
2026-06-20T06:14:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1387 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:14:03 dist-test-slave-zt01 krb5kdc[28896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:14:03 dist-test-slave-zt01 krb5kdc[28896](info): set up 2 sockets
Jun 20 06:14:03 dist-test-slave-zt01 krb5kdc[28896](info): commencing operation
krb5kdc: starting...
W20260620 06:14:04.835289 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:14:04 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:14:04.951519 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33157
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:05.053984 28909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:05.054243 28909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:05.054313 28909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:05.058275 28909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:05.058358 28909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:05.058396 28909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:05.058454 28909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:05.058493 28909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:05.063542 28909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:05.064416 28909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:05.065306 28909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:05.073202 28914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.073231 28915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.073230 28917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:05.073323 28909 server_base.cc:1034] running on GCE node
I20260620 06:14:05.073906 28909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:05.074877 28909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:05.076051 28909 hybrid_clock.cc:648] HybridClock initialized: now 1781936045076023 us; error 46 us; skew 500 ppm
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:05.083671 28909 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:05.084542 28909 webserver.cc:460] Webserver started at http://127.14.209.190:41151/ using document root <none> and password file <none>
I20260620 06:14:05.085021 28909 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:05.085079 28909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:05.085229 28909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:05.087164 28909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "e0d68b9c163d4697a9f33763833e9221"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "aecb9b78b1b1c861be859c03103ad402"
server_key_iv: "f92e138608234b412064b38309858a57"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.087670 28909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "e0d68b9c163d4697a9f33763833e9221"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "aecb9b78b1b1c861be859c03103ad402"
server_key_iv: "f92e138608234b412064b38309858a57"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.091033 28909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:14:05.093238 28924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.094357 28909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:14:05.094501 28909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "e0d68b9c163d4697a9f33763833e9221"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "aecb9b78b1b1c861be859c03103ad402"
server_key_iv: "f92e138608234b412064b38309858a57"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.094614 28909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:05.112329 28909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:05.115157 28909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:05.115347 28909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:05.121906 28909 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:33157
I20260620 06:14:05.122167 28976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:33157 every 8 connection(s)
I20260620 06:14:05.122803 28909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:05.125322 28977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:05.126554 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 28909
I20260620 06:14:05.126698 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:14:05.126981 15174 external_mini_cluster.cc:1428] Setting key 84e1b1529b9be24b94afb6293a10fe28
I20260620 06:14:05.131026 28977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap starting.
I20260620 06:14:05.133436 28977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:05.134297 28977 log.cc:824] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:05.136178 28977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: No bootstrap required, opened a new log
I20260620 06:14:05.140681 28977 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:05.140880 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.128216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.14.209.190:33157)
0620 06:14:05.128594 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.128612 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.128646 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:14:05.129224 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.129238 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.129518 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.129778 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:05.130654 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.131697 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.132860 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.133291 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.133401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.136100 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:05.136106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:05.136120 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:05.136146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:05.137979 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.138395 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.138403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.138421 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.138568 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.138817 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.138822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.138825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.139287 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:05.139493 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.139736 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.140018 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":131,"threads_started":1}
I20260620 06:14:05.140931 28977 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:05.140982 28977 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d68b9c163d4697a9f33763833e9221, State: Initialized, Role: FOLLOWER
I20260620 06:14:05.141410 28977 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:05.141558 28977 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:14:05.141620 28977 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:14:05.141711 28977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:05.142480 28977 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:05.142822 28977 leader_election.cc:302] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d68b9c163d4697a9f33763833e9221; no voters:
I20260620 06:14:05.143167 28977 leader_election.cc:288] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:14:05.143263 28982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:05.143518 28982 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 LEADER]: Becoming Leader. State: Replica: e0d68b9c163d4697a9f33763833e9221, State: Running, Role: LEADER
I20260620 06:14:05.143792 28982 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:05.144351 28977 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:05.145430 28984 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d68b9c163d4697a9f33763833e9221. Latest consensus state: current_term: 1 leader_uuid: "e0d68b9c163d4697a9f33763833e9221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } } }
I20260620 06:14:05.145551 28984 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:05.145824 28983 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d68b9c163d4697a9f33763833e9221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } } }
I20260620 06:14:05.145918 28983 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:05.146559 28991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:05.149415 28991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:05.154770 28991 catalog_manager.cc:1347] Generated new cluster ID: d6e914e4b0cb4045be66a443eeb78b46
I20260620 06:14:05.154829 28991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:14:05.168710 28991 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:14:05.169952 28991 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:14:05.180351 28991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Generated new TSK 0
I20260620 06:14:05.180976 28991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:14:05.210557 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:33157
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:05.314843 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:05.315125 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:05.315222 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:05.319257 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:05.319306 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:05.319362 29003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:14:05.324285 29003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:05.325148 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:05.326100 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:05.334995 29009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.335000 29011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.335004 29008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:05.335043 29003 server_base.cc:1034] running on GCE node
I20260620 06:14:05.335580 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:05.336205 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:05.337389 29003 hybrid_clock.cc:648] HybridClock initialized: now 1781936045337364 us; error 40 us; skew 500 ppm
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:05.340020 29003 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:14:05.340761 29003 webserver.cc:460] Webserver started at http://127.14.209.129:39311/ using document root <none> and password file <none>
I20260620 06:14:05.341269 29003 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:05.341355 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:05.341572 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:05.343637 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "9250fa488cb4460dac67bb7338dfb972"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "047b88e74dcd07d1173de5f7691ce35f"
server_key_iv: "ebc16444566dbcdb9c25201937474caa"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.344189 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "9250fa488cb4460dac67bb7338dfb972"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "047b88e74dcd07d1173de5f7691ce35f"
server_key_iv: "ebc16444566dbcdb9c25201937474caa"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.347860 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:05.350149 29018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.351282 29003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:05.351516 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "9250fa488cb4460dac67bb7338dfb972"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "047b88e74dcd07d1173de5f7691ce35f"
server_key_iv: "ebc16444566dbcdb9c25201937474caa"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.351634 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:05.365428 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:05.368351 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:05.368561 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:05.369083 29003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:05.369858 29003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:05.369904 29003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.369968 29003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:05.370000 29003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.378531 29003 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:45869
I20260620 06:14:05.378562 29131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:45869 every 8 connection(s)
I20260620 06:14:05.379370 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:14:05.386627 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29003
I20260620 06:14:05.386765 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:14:05.387068 15174 external_mini_cluster.cc:1428] Setting key 2e51a2cd67e72dfb3d17cfdd4336c975
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:05.392925 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.380956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:43673 (local address 127.14.209.190:33157)
0620 06:14:05.381080 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.381084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.381918 (+ 834us) server_negotiation.cc:407] Connection header received
0620 06:14:05.383215 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.383218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.383273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.383376 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:05.384409 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.385015 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.386118 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.386364 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.386479 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.389093 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:05.389097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:05.389099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:05.389123 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:05.390798 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.391327 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.391334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.391339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.391429 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.391738 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.391744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.391748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.391996 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:05.392098 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.392598 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.392729 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:14:05.393447 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.381275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:43673)
0620 06:14:05.381774 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.381814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.383004 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.383538 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.383545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.383856 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:05.384266 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.384277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.385152 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.385155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.385992 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.386001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.386529 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.386533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.386706 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.387351 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:05.387366 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:05.388945 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:05.390917 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:05.390921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:05.390931 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:05.391233 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:05.391520 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:05.391522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:05.391523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:05.391639 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:05.392093 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:05.392097 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:05.392320 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.392490 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.392704 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":105,"threads_started":1}
I20260620 06:14:05.394517 29132 heartbeater.cc:347] Connected to a master server at 127.14.209.190:33157
I20260620 06:14:05.394732 29132 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:05.395206 29132 heartbeater.cc:511] Master 127.14.209.190:33157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:14:05.396581 28941 ts_manager.cc:194] Registered new tserver with Master: 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869)
I20260620 06:14:05.397751 28941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:43673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:14:05.413725 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:33157
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:05.520496 29137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:05.520704 29137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:05.520743 29137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:05.524490 29137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:05.524530 29137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:05.524585 29137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:14:05.529461 29137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:05.530252 29137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:05.531124 29137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:05.539718 29143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.539824 29145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.539736 29142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:05.540139 29137 server_base.cc:1034] running on GCE node
I20260620 06:14:05.540426 29137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:05.540920 29137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:05.542116 29137 hybrid_clock.cc:648] HybridClock initialized: now 1781936045542093 us; error 42 us; skew 500 ppm
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:05.544862 29137 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:14:05.545711 29137 webserver.cc:460] Webserver started at http://127.14.209.130:42589/ using document root <none> and password file <none>
I20260620 06:14:05.546211 29137 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:05.546281 29137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:05.546482 29137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:05.548274 29137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "5cc4e35f3e8e434486496806271de862"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "07ff2d10d7349bbcebedcb70d79438a0"
server_key_iv: "2bf9bcf8a736ed85f84dbaf650bb9f16"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.548735 29137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc4e35f3e8e434486496806271de862"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "07ff2d10d7349bbcebedcb70d79438a0"
server_key_iv: "2bf9bcf8a736ed85f84dbaf650bb9f16"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.551947 29137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:05.554034 29152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.555222 29137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:05.555367 29137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "5cc4e35f3e8e434486496806271de862"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "07ff2d10d7349bbcebedcb70d79438a0"
server_key_iv: "2bf9bcf8a736ed85f84dbaf650bb9f16"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.555469 29137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:05.570294 29137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:05.572851 29137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:05.573031 29137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:05.573540 29137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:05.574240 29137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:05.574283 29137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.574347 29137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:05.574389 29137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.582715 29137 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:33551
I20260620 06:14:05.582760 29265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:33551 every 8 connection(s)
I20260620 06:14:05.583487 29137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:14:05.588917 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29137
I20260620 06:14:05.589076 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:14:05.589331 15174 external_mini_cluster.cc:1428] Setting key 2dd5073afd1eb196c1c7e15afdbe128a
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:05.596176 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.584993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:40913 (local address 127.14.209.190:33157)
0620 06:14:05.585122 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.585127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.585874 (+ 747us) server_negotiation.cc:407] Connection header received
0620 06:14:05.587104 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.587108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.587163 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.587233 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:05.588217 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.588719 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.589962 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.590188 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.590294 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.592639 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:05.592642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:05.592644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:05.592664 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:05.594144 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.594731 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.594736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.594738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.594816 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.595171 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.595176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.595178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.595360 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:05.595465 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.595908 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.596019 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:14:05.596824 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.585347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:40913)
0620 06:14:05.585740 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.585777 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.586886 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.587370 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.587376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.587680 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:05.588094 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.588106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.588861 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.588864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.589829 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.589845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.590326 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.590330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.590492 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.590974 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:05.590989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:05.592480 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:05.594268 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:05.594273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:05.594284 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:05.594632 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:05.594913 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:05.594917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:05.594919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:05.595091 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:05.595463 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:05.595469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:05.595680 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.595849 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.596120 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260620 06:14:05.597921 29266 heartbeater.cc:347] Connected to a master server at 127.14.209.190:33157
I20260620 06:14:05.598129 29266 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:14:05.598557 29266 heartbeater.cc:511] Master 127.14.209.190:33157 requested a full tablet report, sending...
I20260620 06:14:05.599531 28941 ts_manager.cc:194] Registered new tserver with Master: 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551)
I20260620 06:14:05.600049 28941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:40913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:14:05.615520 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:33157
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:05.720186 29271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:05.720391 29271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:05.720459 29271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:05.724334 29271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:05.724380 29271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:05.724471 29271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:14:05.729346 29271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:05.730168 29271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:05.731037 29271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:05.739603 29277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.739614 29276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:05.739634 29279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:05.739912 29271 server_base.cc:1034] running on GCE node
I20260620 06:14:05.740280 29271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:05.740832 29271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:05.741986 29271 hybrid_clock.cc:648] HybridClock initialized: now 1781936045741959 us; error 38 us; skew 500 ppm
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:05.744519 29271 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:14:05.745359 29271 webserver.cc:460] Webserver started at http://127.14.209.131:34419/ using document root <none> and password file <none>
I20260620 06:14:05.745849 29271 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:05.745930 29271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:05.746150 29271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:05.747987 29271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "80593aa0b16c4f098342e30662da17e2"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "f2283f4d01d178c0bfad6790d1aedf5c"
server_key_iv: "2e4eb93dd89c64ab9e9536d0a6d1237b"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.748502 29271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "80593aa0b16c4f098342e30662da17e2"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "f2283f4d01d178c0bfad6790d1aedf5c"
server_key_iv: "2e4eb93dd89c64ab9e9536d0a6d1237b"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.751761 29271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:05.753854 29286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.755299 29271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:05.755416 29271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "80593aa0b16c4f098342e30662da17e2"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "f2283f4d01d178c0bfad6790d1aedf5c"
server_key_iv: "2e4eb93dd89c64ab9e9536d0a6d1237b"
server_key_version: "encryptionkey@0"
I20260620 06:14:05.755527 29271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:05.765383 29271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:05.767943 29271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:05.768129 29271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:05.768627 29271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:05.769335 29271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:05.769404 29271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.769465 29271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:05.769508 29271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:05.777768 29271 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:44709
I20260620 06:14:05.777793 29399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:44709 every 8 connection(s)
I20260620 06:14:05.778508 29271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:14:05.780449 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29271
I20260620 06:14:05.780576 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:14:05.780809 15174 external_mini_cluster.cc:1428] Setting key d80215672bfb52ea95874dbafb84f576
Jun 20 06:14:05 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936045, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:05.790562 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.780163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:56647 (local address 127.14.209.190:33157)
0620 06:14:05.780321 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.780328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.781054 (+ 726us) server_negotiation.cc:407] Connection header received
0620 06:14:05.782512 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.782516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.782571 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.782656 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:05.783734 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.784257 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.785277 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.785502 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.785579 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.787671 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:05.787674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:05.787676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:05.787696 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:05.788897 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.789366 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.789369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.789370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.789445 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.789729 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.789732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.789733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.789888 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:05.789962 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.790370 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.790453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:14:05.791347 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.780460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:56647)
0620 06:14:05.780909 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.780954 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.782305 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.782832 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.782840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.783205 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:05.783617 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.783629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.784359 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.784361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.785175 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.785184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.785619 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.785622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.785747 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.786185 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:05.786199 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:05.787535 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:05.789002 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:05.789006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:05.789016 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:05.789248 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:05.789535 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:05.789537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:05.789538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:05.789653 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:05.789971 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:05.789975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:05.790164 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.790300 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.790500 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260620 06:14:05.792383 29400 heartbeater.cc:347] Connected to a master server at 127.14.209.190:33157
I20260620 06:14:05.792598 29400 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:05.793071 29400 heartbeater.cc:511] Master 127.14.209.190:33157 requested a full tablet report, sending...
I20260620 06:14:05.794059 28941 ts_manager.cc:194] Registered new tserver with Master: 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
I20260620 06:14:05.794534 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:14:05.794744 28941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:56647
I20260620 06:14:05.803382 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.796580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.14.209.190:33157)
0620 06:14:05.796685 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.796688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.796897 (+ 209us) server_negotiation.cc:407] Connection header received
0620 06:14:05.796971 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.796972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.797004 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.797067 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:05.797646 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.798184 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.799110 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.799308 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.799384 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.800679 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:05.800682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:05.800683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:05.800703 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:05.801945 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.802304 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.802306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.802307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.802373 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:05.802631 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:05.802633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:05.802635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:05.802809 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:05.802902 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.803204 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.803286 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:14:05.805855 28941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:14:05.808736 28941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:14:05.822906 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.817492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.14.209.130:33551)
0620 06:14:05.818061 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.818077 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.818095 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:14:05.818184 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.818188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.818360 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.818548 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:05.818855 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.820015 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.821702 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.822287 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.822420 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.822551 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.822679 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.822763 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":110,"threads_started":1}
I20260620 06:14:05.822896 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.816943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33551 (local address 127.0.0.1:49358)
0620 06:14:05.817527 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.817584 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.817678 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.818511 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.818515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.818527 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:05.818691 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.818703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.820157 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.820160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.821567 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.821576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.822404 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.822408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.822509 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.822544 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.822653 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.822759 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":100,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20260620 06:14:05.823127 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.817035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:44709 (local address 127.0.0.1:50950)
0620 06:14:05.818797 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.818808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.818914 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.819331 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.819334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.819350 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:05.819442 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.819449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.820440 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.820442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.821505 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.821514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.822704 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.822707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.822787 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.822808 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.822880 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.822945 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260620 06:14:05.823540 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.817443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.14.209.131:44709)
0620 06:14:05.817851 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.817862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.818896 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 06:14:05.819015 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.819025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.819206 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.819339 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:05.819536 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.820329 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.821634 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.822396 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.822528 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.822637 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.823282 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.823382 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260620 06:14:05.823836 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.816878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.14.209.129:45869)
0620 06:14:05.817897 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.817908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.817929 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:14:05.818035 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.818041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.818231 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.818382 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:05.818786 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.820016 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.821832 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.822618 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.822728 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.822836 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.823596 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.823690 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":112,"threads_started":1}
I20260620 06:14:05.824059 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.816717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45869 (local address 127.0.0.1:50412)
0620 06:14:05.817438 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.817483 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.817605 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.818464 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.818469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.818492 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:05.818657 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.818666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.820642 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.820645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.821722 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.821729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.822894 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.822897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.823447 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.823470 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.823555 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.823780 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260620 06:14:05.826094 29334 tablet_service.cc:1467] Processing CreateTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce (DEFAULT_TABLE table=test-table [id=5b14866643134da09e74ada7c6304a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:05.826102 29200 tablet_service.cc:1467] Processing CreateTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce (DEFAULT_TABLE table=test-table [id=5b14866643134da09e74ada7c6304a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:05.827164 29200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e71ffea14d41b98a64a6ec0be5c6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:05.827167 29334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e71ffea14d41b98a64a6ec0be5c6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:05.828459 29066 tablet_service.cc:1467] Processing CreateTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce (DEFAULT_TABLE table=test-table [id=5b14866643134da09e74ada7c6304a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:05.829213 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e71ffea14d41b98a64a6ec0be5c6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:05.834092 29417 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Bootstrap starting.
I20260620 06:14:05.834486 29418 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Bootstrap starting.
I20260620 06:14:05.835196 29419 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Bootstrap starting.
I20260620 06:14:05.837129 29419 tablet_bootstrap.cc:654] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:05.837345 29418 tablet_bootstrap.cc:654] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:05.837342 29417 tablet_bootstrap.cc:654] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:05.837929 29419 log.cc:824] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:05.838133 29417 log.cc:824] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:05.838264 29418 log.cc:824] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:05.839951 29417 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: No bootstrap required, opened a new log
I20260620 06:14:05.839951 29419 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: No bootstrap required, opened a new log
I20260620 06:14:05.839977 29418 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: No bootstrap required, opened a new log
I20260620 06:14:05.840132 29417 ts_tablet_manager.cc:1397] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:14:05.840132 29419 ts_tablet_manager.cc:1397] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:14:05.840157 29418 ts_tablet_manager.cc:1397] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:14:05.842741 29418 raft_consensus.cc:348] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.842962 29418 raft_consensus.cc:374] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:05.843010 29418 raft_consensus.cc:729] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc4e35f3e8e434486496806271de862, State: Initialized, Role: FOLLOWER
I20260620 06:14:05.843427 29418 consensus_queue.cc:260] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.843781 29419 raft_consensus.cc:348] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.843781 29417 raft_consensus.cc:348] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.844005 29266 heartbeater.cc:503] Master 127.14.209.190:33157 was elected leader, sending a full tablet report...
I20260620 06:14:05.844050 29419 raft_consensus.cc:374] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:05.844050 29417 raft_consensus.cc:374] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:05.844101 29417 raft_consensus.cc:729] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80593aa0b16c4f098342e30662da17e2, State: Initialized, Role: FOLLOWER
I20260620 06:14:05.844101 29419 raft_consensus.cc:729] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250fa488cb4460dac67bb7338dfb972, State: Initialized, Role: FOLLOWER
I20260620 06:14:05.844439 29418 ts_tablet_manager.cc:1428] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260620 06:14:05.844515 29417 consensus_queue.cc:260] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.844643 29419 consensus_queue.cc:260] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.845155 29400 heartbeater.cc:503] Master 127.14.209.190:33157 was elected leader, sending a full tablet report...
I20260620 06:14:05.845481 29417 ts_tablet_manager.cc:1428] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:14:05.845481 29419 ts_tablet_manager.cc:1428] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 06:14:05.846786 29132 heartbeater.cc:503] Master 127.14.209.190:33157 was elected leader, sending a full tablet report...
I20260620 06:14:05.870867 29425 raft_consensus.cc:482] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:05.870985 29425 raft_consensus.cc:504] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.871985 29425 leader_election.cc:288] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
I20260620 06:14:05.876451 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.872368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:59093 (local address 127.14.209.131:44709)
0620 06:14:05.872478 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.872482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.872660 (+ 178us) server_negotiation.cc:407] Connection header received
0620 06:14:05.872761 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.872763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.872796 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.872880 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:05.873101 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.873775 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.875212 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.875815 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.875934 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.875981 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.876270 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.876334 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:14:05.876451 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.872253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:55753 (local address 127.14.209.130:33551)
0620 06:14:05.872371 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.872376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.872389 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:14:05.872433 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.872439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.872485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.872560 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:05.872874 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.873634 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.875187 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.875815 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.875934 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.875981 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.876270 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.876334 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:14:05.876519 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.872140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33551 (local address 127.14.209.129:55753)
0620 06:14:05.872252 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.872268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.872344 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.872593 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.872598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.872621 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:05.872768 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.872773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.873736 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.873738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.875043 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.875076 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.875931 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.875935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.876066 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.876159 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.876238 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.876361 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":103,"spinlock_wait_cycles":14336}
I20260620 06:14:05.876716 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.872253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:44709 (local address 127.14.209.129:59093)
0620 06:14:05.872559 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.872571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:05.872672 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:05.872900 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:05.872902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:05.872916 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:05.872996 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.873000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.873867 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.873870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.875110 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:05.875120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.875931 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.875935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:05.876079 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.876159 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 06:14:05.876238 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:14:05.876361 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":141,"spinlock_wait_cycles":384,"thread_start_us":67,"threads_started":1}
I20260620 06:14:05.877139 29220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e71ffea14d41b98a64a6ec0be5c6ce" candidate_uuid: "9250fa488cb4460dac67bb7338dfb972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc4e35f3e8e434486496806271de862" is_pre_election: true
I20260620 06:14:05.877244 29354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e71ffea14d41b98a64a6ec0be5c6ce" candidate_uuid: "9250fa488cb4460dac67bb7338dfb972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80593aa0b16c4f098342e30662da17e2" is_pre_election: true
I20260620 06:14:05.877414 29220 raft_consensus.cc:2454] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250fa488cb4460dac67bb7338dfb972 in term 0.
I20260620 06:14:05.877517 29354 raft_consensus.cc:2454] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9250fa488cb4460dac67bb7338dfb972 in term 0.
I20260620 06:14:05.877890 29022 leader_election.cc:302] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc4e35f3e8e434486496806271de862, 9250fa488cb4460dac67bb7338dfb972; no voters:
I20260620 06:14:05.878140 29425 raft_consensus.cc:2781] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:14:05.878213 29425 raft_consensus.cc:482] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:14:05.878254 29425 raft_consensus.cc:3037] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:05.879204 29425 raft_consensus.cc:504] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.879554 29425 leader_election.cc:288] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 1 election: Requested vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
I20260620 06:14:05.879813 29220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e71ffea14d41b98a64a6ec0be5c6ce" candidate_uuid: "9250fa488cb4460dac67bb7338dfb972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc4e35f3e8e434486496806271de862"
I20260620 06:14:05.879868 29354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e71ffea14d41b98a64a6ec0be5c6ce" candidate_uuid: "9250fa488cb4460dac67bb7338dfb972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80593aa0b16c4f098342e30662da17e2"
I20260620 06:14:05.879941 29220 raft_consensus.cc:3037] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:05.879993 29354 raft_consensus.cc:3037] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 06:14:05.880353 29133 tablet.cc:2369] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:14:05.880930 29220 raft_consensus.cc:2454] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250fa488cb4460dac67bb7338dfb972 in term 1.
I20260620 06:14:05.881150 29354 raft_consensus.cc:2454] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9250fa488cb4460dac67bb7338dfb972 in term 1.
I20260620 06:14:05.881193 29022 leader_election.cc:302] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc4e35f3e8e434486496806271de862, 9250fa488cb4460dac67bb7338dfb972; no voters:
I20260620 06:14:05.881333 29425 raft_consensus.cc:2781] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:05.881538 29425 raft_consensus.cc:686] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 LEADER]: Becoming Leader. State: Replica: 9250fa488cb4460dac67bb7338dfb972, State: Running, Role: LEADER
I20260620 06:14:05.881826 29425 consensus_queue.cc:237] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:05.884640 28939 catalog_manager.cc:5495] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129). New cstate: current_term: 1 leader_uuid: "9250fa488cb4460dac67bb7338dfb972" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } health_report { overall_health: UNKNOWN } } }
I20260620 06:14:05.900604 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.897377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.14.209.129:45869)
0620 06:14:05.897493 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:05.897496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:05.897591 (+ 95us) server_negotiation.cc:407] Connection header received
0620 06:14:05.897721 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:05.897723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:05.897768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:05.897903 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:14:05.898107 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.898567 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.899429 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:05.899636 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:05.899713 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:05.899931 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:14:05.900259 (+ 328us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:14:05.900343 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:05.900397 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 06:14:05.900437 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:14:05.905432 29220 raft_consensus.cc:1261] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Refusing update from remote peer 9250fa488cb4460dac67bb7338dfb972: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:05.905437 29354 raft_consensus.cc:1261] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Refusing update from remote peer 9250fa488cb4460dac67bb7338dfb972: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:05.906059 29427 consensus_queue.cc:1035] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:05.906244 29425 consensus_queue.cc:1035] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:05.912307 29430 mvcc.cc:205] Tried to move back new op lower bound from 7298810044022685696 to 7298810043934191616. Current Snapshot: MvccSnapshot[applied={T|T < 7298810044022685696}]
I20260620 06:14:05.912580 29431 mvcc.cc:205] Tried to move back new op lower bound from 7298810044022685696 to 7298810043934191616. Current Snapshot: MvccSnapshot[applied={T|T < 7298810044022685696}]
I20260620 06:14:05.914621 29432 mvcc.cc:205] Tried to move back new op lower bound from 7298810044022685696 to 7298810043934191616. Current Snapshot: MvccSnapshot[applied={T|T < 7298810044022685696}]
I20260620 06:14:05.919029 28940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:05.919193 28940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:05.921275 28940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Sending DeleteTablet for 3 replicas of tablet 16e71ffea14d41b98a64a6ec0be5c6ce
I20260620 06:14:05.921914 29066 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:05 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:50412
I20260620 06:14:05.921941 29200 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:05 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:49358
I20260620 06:14:05.922058 29334 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:05 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:50950
I20260620 06:14:05.922451 29441 tablet_replica.cc:330] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: stopping tablet replica
I20260620 06:14:05.922544 29439 tablet_replica.cc:330] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: stopping tablet replica
I20260620 06:14:05.922562 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29003
I20260620 06:14:05.923231 29440 tablet_replica.cc:330] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: stopping tablet replica
I20260620 06:14:05.923535 29440 raft_consensus.cc:2229] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:05.923836 29440 raft_consensus.cc:2258] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:05.924270 29441 raft_consensus.cc:2229] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:05.924507 29441 raft_consensus.cc:2258] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:05.925115 29440 ts_tablet_manager.cc:1905] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:05.925587 29441 ts_tablet_manager.cc:1905] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:05.927559 29441 ts_tablet_manager.cc:1918] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:05.927582 29440 ts_tablet_manager.cc:1918] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:05.927656 29440 log.cc:1196] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/16e71ffea14d41b98a64a6ec0be5c6ce
I20260620 06:14:05.927661 29441 log.cc:1196] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/16e71ffea14d41b98a64a6ec0be5c6ce
I20260620 06:14:05.927937 29441 ts_tablet_manager.cc:1939] T 16e71ffea14d41b98a64a6ec0be5c6ce P 80593aa0b16c4f098342e30662da17e2: Deleting consensus metadata
I20260620 06:14:05.927963 29440 ts_tablet_manager.cc:1939] T 16e71ffea14d41b98a64a6ec0be5c6ce P 5cc4e35f3e8e434486496806271de862: Deleting consensus metadata
I20260620 06:14:05.928894 28928 catalog_manager.cc:4841] TS 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): tablet 16e71ffea14d41b98a64a6ec0be5c6ce (table test-table [id=5b14866643134da09e74ada7c6304a7a]) successfully deleted
I20260620 06:14:05.928894 28925 catalog_manager.cc:4841] TS 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): tablet 16e71ffea14d41b98a64a6ec0be5c6ce (table test-table [id=5b14866643134da09e74ada7c6304a7a]) successfully deleted
W20260620 06:14:05.929383 28927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:45869 (error 108)
I20260620 06:14:05.930094 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29137
I20260620 06:14:05.930286 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:05.929987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45869 (local address 127.0.0.1:50432)
0620 06:14:05.930121 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:05.930198 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 06:14:05.930727 28927 catalog_manager.cc:4584] TS 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16e71ffea14d41b98a64a6ec0be5c6ce: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45869: connect: Connection refused (error 111)
I20260620 06:14:05.935675 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29271
I20260620 06:14:05.941069 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 28909
I20260620 06:14:05.946964 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33157
--webserver_interface=127.14.209.190
--webserver_port=41151
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:06.049921 29443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:06.050134 29443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:06.050199 29443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:06.054108 29443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:06.054155 29443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:06.054174 29443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:06.054190 29443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:06.054245 29443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:06.059178 29443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:06.060006 29443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:06.060866 29443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:06.072801 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.072800 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.072795 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:06.073143 29443 server_base.cc:1034] running on GCE node
I20260620 06:14:06.073729 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:06.074730 29443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:06.075971 29443 hybrid_clock.cc:648] HybridClock initialized: now 1781936046075939 us; error 44 us; skew 500 ppm
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:06.078686 29443 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:06.079634 29443 webserver.cc:460] Webserver started at http://127.14.209.190:41151/ using document root <none> and password file <none>
I20260620 06:14:06.080176 29443 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:06.080278 29443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:06.083956 29443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:06.085934 29459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.087047 29443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:06.087251 29443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "e0d68b9c163d4697a9f33763833e9221"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "aecb9b78b1b1c861be859c03103ad402"
server_key_iv: "f92e138608234b412064b38309858a57"
server_key_version: "encryptionkey@0"
I20260620 06:14:06.087826 29443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:06.109781 29443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:06.112672 29443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:06.112874 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:06.119892 29443 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:33157
I20260620 06:14:06.120159 29511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:33157 every 8 connection(s)
I20260620 06:14:06.120998 29443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:06.122464 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29443
I20260620 06:14:06.122936 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:45869
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:33157
--webserver_port=39311
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:06.125722 29512 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:14:06.127383 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap starting.
I20260620 06:14:06.143401 29512 log.cc:824] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:06.151018 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:14:06.151356 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap complete.
I20260620 06:14:06.155139 29512 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:06.155541 29512 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d68b9c163d4697a9f33763833e9221, State: Initialized, Role: FOLLOWER
I20260620 06:14:06.156020 29512 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:06.156157 29512 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:14:06.156244 29512 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:14:06.156355 29512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:14:06.158097 29512 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:06.158404 29512 leader_election.cc:302] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d68b9c163d4697a9f33763833e9221; no voters:
I20260620 06:14:06.158684 29512 leader_election.cc:288] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:14:06.158815 29516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:14:06.159081 29516 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 LEADER]: Becoming Leader. State: Replica: e0d68b9c163d4697a9f33763833e9221, State: Running, Role: LEADER
I20260620 06:14:06.159368 29516 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:06.159775 29512 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:06.160657 29518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d68b9c163d4697a9f33763833e9221. Latest consensus state: current_term: 2 leader_uuid: "e0d68b9c163d4697a9f33763833e9221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } } }
I20260620 06:14:06.160702 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0d68b9c163d4697a9f33763833e9221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } } }
I20260620 06:14:06.160789 29518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:06.160795 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:06.161212 29520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:06.165767 29520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:06.166353 29520 catalog_manager.cc:1259] Loaded cluster ID: d6e914e4b0cb4045be66a443eeb78b46
I20260620 06:14:06.166412 29520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:14:06.167812 29520 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 06:14:06.174101 29520 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 06:14:06.179127 29520 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:14:06.179785 29520 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Loaded TSK: 0
I20260620 06:14:06.180892 29520 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:14:06.267643 29514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:06.267972 29514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:06.268049 29514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:06.275913 29514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:06.275962 29514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:06.276049 29514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:14:06.281067 29514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=39311
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:06.281975 29514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:06.282888 29514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:06.292061 29540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.292052 29539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.292032 29542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:06.292449 29514 server_base.cc:1034] running on GCE node
I20260620 06:14:06.292843 29514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:06.293458 29514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:06.294642 29514 hybrid_clock.cc:648] HybridClock initialized: now 1781936046294619 us; error 45 us; skew 500 ppm
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:06.297472 29514 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:14:06.298393 29514 webserver.cc:460] Webserver started at http://127.14.209.129:39311/ using document root <none> and password file <none>
I20260620 06:14:06.298915 29514 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:06.299018 29514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:06.302640 29514 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:06.304724 29549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.305910 29514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:14:06.306020 29514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "9250fa488cb4460dac67bb7338dfb972"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "047b88e74dcd07d1173de5f7691ce35f"
server_key_iv: "ebc16444566dbcdb9c25201937474caa"
server_key_version: "encryptionkey@0"
I20260620 06:14:06.306533 29514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:06.315825 29514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:06.318674 29514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:06.318961 29514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:06.319533 29514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:06.320521 29556 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:14:06.324378 29514 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:14:06.324455 29514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 06:14:06.324534 29514 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:14:06.326054 29514 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:14:06.326119 29514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:14:06.326406 29556 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Bootstrap starting.
I20260620 06:14:06.336272 29514 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:45869
I20260620 06:14:06.336347 29663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:45869 every 8 connection(s)
I20260620 06:14:06.337110 29514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:14:06.339443 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29514
I20260620 06:14:06.339905 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:33551
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:33157
--webserver_port=42589
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:06.344846 29556 log.cc:824] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:06.353899 29556 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:14:06.354291 29556 tablet_bootstrap.cc:492] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Bootstrap complete.
I20260620 06:14:06.354620 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.339129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:33455)
0620 06:14:06.339867 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.339930 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.341210 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.342599 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.342611 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.343177 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:06.343760 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.343777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.345839 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.345843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.346688 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.346699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.347476 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.347480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.347657 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.348241 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:06.348256 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:06.349630 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:06.351700 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.351705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.351717 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.351974 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.352328 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.352331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.352333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.352468 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.353002 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:06.353008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:06.353231 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.353405 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.353705 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":195,"threads_started":1}
I20260620 06:14:06.354611 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.339118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:33455 (local address 127.14.209.190:33157)
0620 06:14:06.340219 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.340240 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.340286 (+ 46us) server_negotiation.cc:407] Connection header received
0620 06:14:06.341708 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.341723 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.342132 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.342472 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:06.343959 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.345654 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.346826 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.347328 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.347527 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.349822 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:06.349827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:06.349838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:06.349865 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:06.351562 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.352090 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.352095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.352102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.352222 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.352572 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.352575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.352577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.352883 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:06.353025 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.353518 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.353759 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":115,"threads_started":1}
I20260620 06:14:06.355692 29556 ts_tablet_manager.cc:1397] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.004s
I20260620 06:14:06.357129 29664 heartbeater.cc:347] Connected to a master server at 127.14.209.190:33157
I20260620 06:14:06.357340 29664 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:06.357765 29664 heartbeater.cc:511] Master 127.14.209.190:33157 requested a full tablet report, sending...
I20260620 06:14:06.358404 29556 raft_consensus.cc:348] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:06.358883 29556 raft_consensus.cc:729] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250fa488cb4460dac67bb7338dfb972, State: Initialized, Role: FOLLOWER
I20260620 06:14:06.359395 29556 consensus_queue.cc:260] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } } peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } }
I20260620 06:14:06.360023 29476 ts_manager.cc:194] Registered new tserver with Master: 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869)
I20260620 06:14:06.360366 29556 ts_tablet_manager.cc:1428] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:14:06.363406 29476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:33455
I20260620 06:14:06.366171 29664 heartbeater.cc:503] Master 127.14.209.190:33157 was elected leader, sending a full tablet report...
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:14:06.373202 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.362851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.14.209.129:45869)
0620 06:14:06.363179 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.363189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.363530 (+ 341us) server_negotiation.cc:407] Connection header received
0620 06:14:06.363790 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.363796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.363987 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.364102 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:06.364903 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.365931 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.367111 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.367619 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.367721 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.369890 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:06.369897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:06.369905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:06.369925 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:06.371476 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.372008 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.372012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.372013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.372082 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.372349 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.372351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.372352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.372552 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:06.372647 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.372980 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.373093 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
I20260620 06:14:06.373239 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.362772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45869 (local address 127.0.0.1:50446)
0620 06:14:06.363398 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.363451 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.363617 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.364253 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.364258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.364660 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:06.364780 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.364788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.366059 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.366063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.366968 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.366978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.367716 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.367719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.367830 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.368349 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:06.368361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:06.369748 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:06.371588 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.371600 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.371608 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.371889 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.372168 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.372171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.372173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.372265 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.372657 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:06.372663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:06.372775 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.372920 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.373084 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":86,"threads_started":1}
I20260620 06:14:06.374418 29598 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e71ffea14d41b98a64a6ec0be5c6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:05 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:50446
I20260620 06:14:06.375326 29673 tablet_replica.cc:330] stopping tablet replica
I20260620 06:14:06.375597 29673 raft_consensus.cc:2229] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:06.375810 29673 raft_consensus.cc:2258] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:06.376632 29673 ts_tablet_manager.cc:1905] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:06.379799 29673 ts_tablet_manager.cc:1918] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:06.379907 29673 log.cc:1196] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/16e71ffea14d41b98a64a6ec0be5c6ce
I20260620 06:14:06.380168 29673 ts_tablet_manager.cc:1939] T 16e71ffea14d41b98a64a6ec0be5c6ce P 9250fa488cb4460dac67bb7338dfb972: Deleting consensus metadata
I20260620 06:14:06.381584 29462 catalog_manager.cc:4841] TS 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869): tablet 16e71ffea14d41b98a64a6ec0be5c6ce (table test-table [id=5b14866643134da09e74ada7c6304a7a]) successfully deleted
W20260620 06:14:06.453339 29668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:06.453563 29668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:06.453603 29668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:06.457376 29668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:06.457424 29668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:06.457479 29668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:14:06.462591 29668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:33551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=42589
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:06.463472 29668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:06.464309 29668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:06.473001 29678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.473006 29681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.473094 29679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:06.473228 29668 server_base.cc:1034] running on GCE node
I20260620 06:14:06.473733 29668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:06.474321 29668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:06.475472 29668 hybrid_clock.cc:648] HybridClock initialized: now 1781936046475448 us; error 37 us; skew 500 ppm
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:06.478014 29668 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:14:06.478798 29668 webserver.cc:460] Webserver started at http://127.14.209.130:42589/ using document root <none> and password file <none>
I20260620 06:14:06.479364 29668 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:06.479475 29668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:06.482908 29668 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:06.484905 29688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.486037 29668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:06.486198 29668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "5cc4e35f3e8e434486496806271de862"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "07ff2d10d7349bbcebedcb70d79438a0"
server_key_iv: "2bf9bcf8a736ed85f84dbaf650bb9f16"
server_key_version: "encryptionkey@0"
I20260620 06:14:06.486699 29668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:06.497809 29668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:06.500756 29668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:06.500941 29668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:06.501461 29668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:06.502173 29668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:06.502241 29668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.502306 29668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:06.502360 29668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.511809 29668 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:33551
I20260620 06:14:06.511849 29801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:33551 every 8 connection(s)
I20260620 06:14:06.512693 29668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:14:06.516865 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29668
I20260620 06:14:06.517371 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:44709
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:33157
--webserver_port=34419
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:06.527485 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.514310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:46821 (local address 127.14.209.190:33157)
0620 06:14:06.514516 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.514521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.515287 (+ 766us) server_negotiation.cc:407] Connection header received
0620 06:14:06.516675 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.516677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.516724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.516803 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:06.518400 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.519132 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.520641 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.521087 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.521208 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.523560 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:06.523567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:06.523572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:06.523604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:06.525338 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.525849 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.525856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.525858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.525953 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.526274 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.526279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.526281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.526525 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:06.526642 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.527113 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.527255 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 06:14:06.528129 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.514645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:46821)
0620 06:14:06.515155 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.515193 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.516437 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.517036 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.517052 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.517551 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:06.518229 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.518250 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.519267 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.519272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.520515 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.520524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.521234 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.521236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.521378 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.522018 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:06.522033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:06.523394 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:06.525473 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.525477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.525486 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.525738 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.526054 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.526056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.526057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.526177 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.526641 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:06.526646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:06.526862 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.527008 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.527315 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260620 06:14:06.529203 29802 heartbeater.cc:347] Connected to a master server at 127.14.209.190:33157
I20260620 06:14:06.529444 29802 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:06.529891 29802 heartbeater.cc:511] Master 127.14.209.190:33157 requested a full tablet report, sending...
I20260620 06:14:06.530838 29476 ts_manager.cc:194] Registered new tserver with Master: 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551)
I20260620 06:14:06.531426 29476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:46821
W20260620 06:14:06.621850 29805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:06.622081 29805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:06.622148 29805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:06.626046 29805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:06.626122 29805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:06.626214 29805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:14:06.630993 29805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:44709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=34419
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:06.631834 29805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:06.632673 29805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:06.641206 29810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.641207 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.641321 29811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:06.641419 29805 server_base.cc:1034] running on GCE node
I20260620 06:14:06.641891 29805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:06.642452 29805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:06.643638 29805 hybrid_clock.cc:648] HybridClock initialized: now 1781936046643606 us; error 42 us; skew 500 ppm
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:06.646109 29805 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:14:06.646915 29805 webserver.cc:460] Webserver started at http://127.14.209.131:34419/ using document root <none> and password file <none>
I20260620 06:14:06.647481 29805 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:06.647588 29805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:06.650928 29805 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:06.652798 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.653963 29805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:06.654109 29805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "80593aa0b16c4f098342e30662da17e2"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "f2283f4d01d178c0bfad6790d1aedf5c"
server_key_iv: "2e4eb93dd89c64ab9e9536d0a6d1237b"
server_key_version: "encryptionkey@0"
I20260620 06:14:06.654573 29805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:06.674363 29805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:06.677170 29805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:06.677361 29805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:06.677894 29805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:06.678675 29805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:06.678867 29805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.678989 29805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:06.679042 29805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.688341 29805 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:44709
I20260620 06:14:06.688401 29933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:44709 every 8 connection(s)
I20260620 06:14:06.689358 29805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:14:06.693801 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29805
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:06.703143 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.694406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.14.209.190:33157)
0620 06:14:06.694630 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.694634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.694829 (+ 195us) server_negotiation.cc:407] Connection header received
0620 06:14:06.694924 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.694927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.694971 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.695098 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:06.695932 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.696567 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.697685 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.698022 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.698107 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.699758 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:06.699764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:06.699771 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:06.699792 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:06.701487 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.701906 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.701912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.701914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.701996 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.702267 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.702272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.702274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.702449 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:06.702664 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.702797 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.702902 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"spinlock_wait_cycles":2304,"thread_start_us":83,"threads_started":1}
I20260620 06:14:06.703858 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.691125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:46997 (local address 127.14.209.190:33157)
0620 06:14:06.691232 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.691236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.692024 (+ 788us) server_negotiation.cc:407] Connection header received
0620 06:14:06.693364 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.693367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.693420 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.693536 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:06.694763 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.695535 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.696786 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.696985 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.697061 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.700458 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:06.700462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:06.700463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:06.700499 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:06.701910 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.702390 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.702395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.702397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.702483 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:06.702821 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:06.702825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:06.702827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:06.703012 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:06.703166 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.703608 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.703713 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29,"spinlock_wait_cycles":768}
I20260620 06:14:06.704646 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.691486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:46997)
0620 06:14:06.691887 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.691925 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.693175 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.693684 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.693692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.694080 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:06.694622 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.694642 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.695652 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.695656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.696674 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.696686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.697117 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.697120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.697293 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.697990 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:06.698023 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:06.700295 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:06.702033 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.702039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.702051 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.702288 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.702583 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:06.702587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:06.702589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:06.702722 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:06.703160 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:06.703167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:06.703410 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.703583 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.703855 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260620 06:14:06.705579 29934 heartbeater.cc:347] Connected to a master server at 127.14.209.190:33157
I20260620 06:14:06.705797 29934 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:06.706234 29934 heartbeater.cc:511] Master 127.14.209.190:33157 requested a full tablet report, sending...
I20260620 06:14:06.707195 29476 ts_manager.cc:194] Registered new tserver with Master: 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
I20260620 06:14:06.707721 29476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:46997
I20260620 06:14:06.716676 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:14:06.720786 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.717227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.14.209.190:33157)
0620 06:14:06.717362 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.717366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.717509 (+ 143us) server_negotiation.cc:407] Connection header received
0620 06:14:06.717679 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.717686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.717769 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.717874 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:14:06.718117 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.718615 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.719511 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.719721 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.719827 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.720129 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:14:06.720437 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:14:06.720561 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.720620 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.720666 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:14:06.721755 29476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:14:06.724870 29476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:14:06.731945 29598 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 (DEFAULT_TABLE table=test-table [id=6b261bb544584dc3b594f9eb163d75d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:06.732570 29598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:06.735325 29556 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Bootstrap starting.
I20260620 06:14:06.735723 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.730858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33551 (local address 127.0.0.1:49364)
0620 06:14:06.730975 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.730989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.731094 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.731883 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.731886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.731901 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:06.731987 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.731992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.733172 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.733178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.734175 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.734184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.735343 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.735348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.735440 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.735483 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.735538 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.735611 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:14:06.736215 29556 tablet_bootstrap.cc:654] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:06.736507 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.731549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.14.209.131:44709)
0620 06:14:06.731915 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.731926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.731948 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:14:06.732015 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.732019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.732182 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.732331 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:06.732534 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.733507 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.735086 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.735828 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.735949 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.736035 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.736292 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.736376 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260620 06:14:06.736703 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.731197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:44709 (local address 127.0.0.1:50960)
0620 06:14:06.731645 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.731657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.731735 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.732321 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.732324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.732342 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:06.732419 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.732424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.733646 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.733649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.734927 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.734940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.736087 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.736089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.736156 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.736171 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.736263 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.736328 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260620 06:14:06.736790 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.731031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.14.209.130:33551)
0620 06:14:06.731423 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.731435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.731456 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:14:06.731537 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.731546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.731747 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.731902 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:06.732093 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.733043 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.734312 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.735191 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.736079 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.736226 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.736337 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.736455 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":152,"threads_started":1}
I20260620 06:14:06.737802 29556 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: No bootstrap required, opened a new log
I20260620 06:14:06.737874 29556 ts_tablet_manager.cc:1397] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:14:06.738262 29556 raft_consensus.cc:348] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.738351 29556 raft_consensus.cc:374] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:06.738379 29556 raft_consensus.cc:729] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250fa488cb4460dac67bb7338dfb972, State: Initialized, Role: FOLLOWER
I20260620 06:14:06.738528 29556 consensus_queue.cc:260] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.738802 29556 ts_tablet_manager.cc:1428] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:14:06.739544 29736 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 (DEFAULT_TABLE table=test-table [id=6b261bb544584dc3b594f9eb163d75d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:06.740581 29736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:06.745853 29868 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 (DEFAULT_TABLE table=test-table [id=6b261bb544584dc3b594f9eb163d75d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:06.746506 29947 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Bootstrap starting.
I20260620 06:14:06.746685 29868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:06.748611 29947 tablet_bootstrap.cc:654] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:06.749471 29947 log.cc:824] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:06.750994 29947 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: No bootstrap required, opened a new log
I20260620 06:14:06.751215 29947 ts_tablet_manager.cc:1397] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:14:06.752740 29949 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Bootstrap starting.
I20260620 06:14:06.754487 29947 raft_consensus.cc:348] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.754748 29947 raft_consensus.cc:374] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:06.754808 29947 raft_consensus.cc:729] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc4e35f3e8e434486496806271de862, State: Initialized, Role: FOLLOWER
I20260620 06:14:06.755187 29949 tablet_bootstrap.cc:654] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:06.755301 29947 consensus_queue.cc:260] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.755923 29802 heartbeater.cc:503] Master 127.14.209.190:33157 was elected leader, sending a full tablet report...
I20260620 06:14:06.755962 29949 log.cc:824] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:06.756294 29947 ts_tablet_manager.cc:1428] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:14:06.757449 29950 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:06.757586 29950 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.758592 29949 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: No bootstrap required, opened a new log
I20260620 06:14:06.758769 29949 ts_tablet_manager.cc:1397] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:14:06.758993 29950 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709), 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869)
W20260620 06:14:06.763502 29803 tablet.cc:2369] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:14:06.763703 29949 raft_consensus.cc:348] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.763959 29949 raft_consensus.cc:374] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:06.764125 29949 raft_consensus.cc:729] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80593aa0b16c4f098342e30662da17e2, State: Initialized, Role: FOLLOWER
I20260620 06:14:06.764547 29949 consensus_queue.cc:260] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.765112 29934 heartbeater.cc:503] Master 127.14.209.190:33157 was elected leader, sending a full tablet report...
I20260620 06:14:06.766379 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.759237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45869 (local address 127.14.209.130:55453)
0620 06:14:06.759490 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.759506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.759588 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.760792 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.760795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.760809 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:06.761383 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.761393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.763560 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.763563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.764844 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.764855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.765958 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.765962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.766078 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.766108 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.766180 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.766245 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 06:14:06.766410 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.760118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:55453 (local address 127.14.209.129:45869)
0620 06:14:06.760411 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.760417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.760433 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:14:06.760508 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.760511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.760595 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.760943 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:06.761588 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.763381 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.765112 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.765833 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.765957 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.766081 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.766196 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.766285 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 06:14:06.766919 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.759636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:34903 (local address 127.14.209.131:44709)
0620 06:14:06.759743 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.759747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.762525 (+ 2778us) server_negotiation.cc:407] Connection header received
0620 06:14:06.762774 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.762777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.762818 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.762892 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:06.763143 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.763813 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.765594 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.766167 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.766273 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.766312 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.766733 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.766806 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:14:06.767383 29618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb17fe4f8e641e1bf16ee7cda604aa6" candidate_uuid: "5cc4e35f3e8e434486496806271de862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250fa488cb4460dac67bb7338dfb972" is_pre_election: true
I20260620 06:14:06.767668 29618 raft_consensus.cc:2454] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cc4e35f3e8e434486496806271de862 in term 0.
I20260620 06:14:06.767860 29949 ts_tablet_manager.cc:1428] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:14:06.768124 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.759313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:44709 (local address 127.14.209.130:34903)
0620 06:14:06.762426 (+ 3113us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.762443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:06.762550 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:06.762904 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:06.762907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:06.762925 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:06.763030 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.763036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.764159 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.764163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.765273 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:06.765283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.766379 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.766382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:06.766565 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.766601 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:14:06.767919 (+ 1318us) client_negotiation.cc:240] Negotiation successful
0620 06:14:06.768000 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3010,"thread_start_us":91,"threads_started":1}
I20260620 06:14:06.768131 29691 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc4e35f3e8e434486496806271de862, 9250fa488cb4460dac67bb7338dfb972; no voters:
I20260620 06:14:06.768399 29950 raft_consensus.cc:2781] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:14:06.768496 29950 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:14:06.768549 29950 raft_consensus.cc:3037] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:06.769724 29950 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.770251 29950 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [CANDIDATE]: Term 1 election: Requested vote from peers 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709), 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869)
I20260620 06:14:06.770569 29618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb17fe4f8e641e1bf16ee7cda604aa6" candidate_uuid: "5cc4e35f3e8e434486496806271de862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9250fa488cb4460dac67bb7338dfb972"
I20260620 06:14:06.770710 29618 raft_consensus.cc:3037] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:06.770991 29887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb17fe4f8e641e1bf16ee7cda604aa6" candidate_uuid: "5cc4e35f3e8e434486496806271de862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80593aa0b16c4f098342e30662da17e2"
I20260620 06:14:06.771087 29888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb17fe4f8e641e1bf16ee7cda604aa6" candidate_uuid: "5cc4e35f3e8e434486496806271de862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80593aa0b16c4f098342e30662da17e2" is_pre_election: true
I20260620 06:14:06.771210 29887 raft_consensus.cc:3037] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:06.771968 29618 raft_consensus.cc:2454] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cc4e35f3e8e434486496806271de862 in term 1.
I20260620 06:14:06.772316 29691 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc4e35f3e8e434486496806271de862, 9250fa488cb4460dac67bb7338dfb972; no voters:
I20260620 06:14:06.772425 29887 raft_consensus.cc:2454] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cc4e35f3e8e434486496806271de862 in term 1.
I20260620 06:14:06.772753 29950 raft_consensus.cc:2781] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:06.772989 29950 raft_consensus.cc:686] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 1 LEADER]: Becoming Leader. State: Replica: 5cc4e35f3e8e434486496806271de862, State: Running, Role: LEADER
I20260620 06:14:06.773291 29950 consensus_queue.cc:237] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:06.776326 29476 catalog_manager.cc:5495] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cc4e35f3e8e434486496806271de862 (127.14.209.130). New cstate: current_term: 1 leader_uuid: "5cc4e35f3e8e434486496806271de862" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } health_report { overall_health: UNKNOWN } } }
I20260620 06:14:06.795229 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:06.790856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.14.209.130:33551)
0620 06:14:06.790988 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:06.790993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:06.791043 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:14:06.791237 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:06.791240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:06.791293 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:06.791389 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:14:06.791640 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.792483 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.793860 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:06.794088 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:06.794196 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:06.794495 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:14:06.794816 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:14:06.794936 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:06.795014 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 06:14:06.795092 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 06:14:06.800526 29887 raft_consensus.cc:1261] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Refusing update from remote peer 5cc4e35f3e8e434486496806271de862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:06.800578 29618 raft_consensus.cc:1261] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Refusing update from remote peer 5cc4e35f3e8e434486496806271de862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:06.801086 29950 consensus_queue.cc:1035] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:06.801288 29954 consensus_queue.cc:1035] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:06.809764 29958 mvcc.cc:205] Tried to move back new op lower bound from 7298810047688937472 to 7298810047585890304. Current Snapshot: MvccSnapshot[applied={T|T < 7298810047688937472}]
I20260620 06:14:06.813838 29960 mvcc.cc:205] Tried to move back new op lower bound from 7298810047688937472 to 7298810047585890304. Current Snapshot: MvccSnapshot[applied={T|T < 7298810047688937472}]
I20260620 06:14:06.814718 29475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:06.814868 29475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:06.816994 29475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Sending DeleteTablet for 3 replicas of tablet 2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:06.817575 29736 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:06 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:49364
I20260620 06:14:06.817725 29868 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:06 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:50960
I20260620 06:14:06.817909 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29514
I20260620 06:14:06.818461 29966 tablet_replica.cc:330] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: stopping tablet replica
I20260620 06:14:06.818809 29966 raft_consensus.cc:2229] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:14:06.819190 29966 raft_consensus.cc:2258] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:06.820147 29967 tablet_replica.cc:330] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: stopping tablet replica
I20260620 06:14:06.820403 29967 raft_consensus.cc:2229] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:06.820681 29967 raft_consensus.cc:2258] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:06.820727 29966 ts_tablet_manager.cc:1905] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:06.824275 29967 ts_tablet_manager.cc:1905] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:06.824573 29966 ts_tablet_manager.cc:1918] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:06.824716 29966 log.cc:1196] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:06.825109 29966 ts_tablet_manager.cc:1939] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 5cc4e35f3e8e434486496806271de862: Deleting consensus metadata
I20260620 06:14:06.826222 29463 catalog_manager.cc:4841] TS 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 (table test-table [id=6b261bb544584dc3b594f9eb163d75d7]) successfully deleted
W20260620 06:14:06.826375 29462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:45869 (error 108)
I20260620 06:14:06.826679 29967 ts_tablet_manager.cc:1918] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:06.826767 29967 log.cc:1196] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:06.827108 29967 ts_tablet_manager.cc:1939] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 80593aa0b16c4f098342e30662da17e2: Deleting consensus metadata
W20260620 06:14:06.827559 29944 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:14:06.827255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45869 (local address 127.0.0.1:50454)
0620 06:14:06.827386 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:06.827458 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45869: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:14:06.827674 29462 catalog_manager.cc:4584] TS 9250fa488cb4460dac67bb7338dfb972 (127.14.209.129:45869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cb17fe4f8e641e1bf16ee7cda604aa6: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45869: connect: Connection reset by peer (error 104)
I20260620 06:14:06.828356 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29668
I20260620 06:14:06.828359 29460 catalog_manager.cc:4841] TS 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): tablet 2cb17fe4f8e641e1bf16ee7cda604aa6 (table test-table [id=6b261bb544584dc3b594f9eb163d75d7]) successfully deleted
I20260620 06:14:06.834028 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29805
I20260620 06:14:06.839318 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 29443
I20260620 06:14:06.845372 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33157
--webserver_interface=127.14.209.190
--webserver_port=41151
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:06.948057 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:06.948259 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:06.948297 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:06.952179 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:06.952225 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:06.952244 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:06.952277 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:06.952312 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:06.957149 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:06.957973 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:06.958828 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:14:06.966636 29969 server_base.cc:1034] running on GCE node
W20260620 06:14:06.966643 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.966652 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:06.966645 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:06.967474 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:06.968407 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:06.969585 29969 hybrid_clock.cc:648] HybridClock initialized: now 1781936046969574 us; error 34 us; skew 500 ppm
Jun 20 06:14:06 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:06.972038 29969 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:06.972782 29969 webserver.cc:460] Webserver started at http://127.14.209.190:41151/ using document root <none> and password file <none>
I20260620 06:14:06.973222 29969 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:06.973276 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:06.976598 29969 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:14:06.978386 29984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:06.979547 29969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:06.979664 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "e0d68b9c163d4697a9f33763833e9221"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "aecb9b78b1b1c861be859c03103ad402"
server_key_iv: "f92e138608234b412064b38309858a57"
server_key_version: "encryptionkey@0"
I20260620 06:14:06.980109 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:07.003890 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:07.006397 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:07.006585 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:07.013094 29969 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:33157
I20260620 06:14:07.013365 30036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:33157 every 8 connection(s)
I20260620 06:14:07.014038 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:07.017751 30037 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:14:07.019285 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap starting.
I20260620 06:14:07.020967 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 29969
I20260620 06:14:07.021442 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:45869
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:33157
--webserver_port=39311
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:07.036317 30037 log.cc:824] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:07.046736 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:14:07.047135 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221: Bootstrap complete.
I20260620 06:14:07.050848 30037 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:07.051258 30037 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d68b9c163d4697a9f33763833e9221, State: Initialized, Role: FOLLOWER
I20260620 06:14:07.051710 30037 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:07.051817 30037 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:14:07.051903 30037 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:14:07.052017 30037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:14:07.053709 30037 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:07.054018 30037 leader_election.cc:302] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d68b9c163d4697a9f33763833e9221; no voters:
I20260620 06:14:07.054297 30037 leader_election.cc:288] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 06:14:07.054425 30041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:14:07.054651 30041 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [term 3 LEADER]: Becoming Leader. State: Replica: e0d68b9c163d4697a9f33763833e9221, State: Running, Role: LEADER
I20260620 06:14:07.054934 30041 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } }
I20260620 06:14:07.055486 30037 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:07.055840 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d68b9c163d4697a9f33763833e9221. Latest consensus state: current_term: 3 leader_uuid: "e0d68b9c163d4697a9f33763833e9221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } } }
I20260620 06:14:07.055950 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:07.056211 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e0d68b9c163d4697a9f33763833e9221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d68b9c163d4697a9f33763833e9221" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33157 } } }
I20260620 06:14:07.056309 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d68b9c163d4697a9f33763833e9221 [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:07.056334 30045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:07.060904 30045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:07.061553 30045 catalog_manager.cc:1259] Loaded cluster ID: d6e914e4b0cb4045be66a443eeb78b46
I20260620 06:14:07.061617 30045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 06:14:07.062553 30045 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6abc7b49ad google::LogMessage::Fail() at ??:0
@ 0x7f6abc7b788a google::LogMessage::SendToLog() at ??:0
@ 0x7f6abc7b44de google::LogMessage::Flush() at ??:0
@ 0x7f6abc7b5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6ac70f3fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6ac70f4d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6ac70f1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6ac7135640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6abea063d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6abea06c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6abea083f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6abe9f7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ac003b6db start_thread at ??:0
@ 0x7f6abb2b261f clone at ??:0
W20260620 06:14:07.163360 30039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:07.163713 30039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:07.163784 30039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:07.170260 30039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:07.170331 30039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:07.170433 30039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:14:07.175796 30039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=39311
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:07.176718 30039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:07.177656 30039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:07.186910 30081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:07.186936 30078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:07.186893 30079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:07.187330 30039 server_base.cc:1034] running on GCE node
I20260620 06:14:07.187973 30039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:07.188593 30039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:07.189767 30039 hybrid_clock.cc:648] HybridClock initialized: now 1781936047189742 us; error 41 us; skew 500 ppm
Jun 20 06:14:07 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936047, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:07.192483 30039 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:14:07.193360 30039 webserver.cc:460] Webserver started at http://127.14.209.129:39311/ using document root <none> and password file <none>
I20260620 06:14:07.193871 30039 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:07.193928 30039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:07.197754 30039 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:14:07.199781 30088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.200927 30039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:14:07.201015 30039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "9250fa488cb4460dac67bb7338dfb972"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "047b88e74dcd07d1173de5f7691ce35f"
server_key_iv: "ebc16444566dbcdb9c25201937474caa"
server_key_version: "encryptionkey@0"
I20260620 06:14:07.201521 30039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:07.221819 30039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:07.224741 30039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:07.224931 30039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:07.225522 30039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:07.226670 30095 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:14:07.231400 30039 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:14:07.231464 30039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 06:14:07.231529 30039 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:14:07.233013 30039 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:14:07.233068 30039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:14:07.233299 30095 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Bootstrap starting.
I20260620 06:14:07.244012 30039 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:45869
I20260620 06:14:07.245209 30039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:14:07.248153 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30039
I20260620 06:14:07.248625 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:33551
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:33157
--webserver_port=42589
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:07.254315 30095 log.cc:824] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:07.254634 30202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:45869 every 8 connection(s)
I20260620 06:14:07.263862 30095 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:14:07.264278 30095 tablet_bootstrap.cc:492] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Bootstrap complete.
I20260620 06:14:07.265708 30095 ts_tablet_manager.cc:1397] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260620 06:14:07.268957 30095 raft_consensus.cc:348] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:07.269479 30095 raft_consensus.cc:729] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9250fa488cb4460dac67bb7338dfb972, State: Initialized, Role: FOLLOWER
I20260620 06:14:07.269963 30095 consensus_queue.cc:260] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:07.270920 30095 ts_tablet_manager.cc:1428] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260620 06:14:07.308447 30205 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:14:07.247581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:51073)
0620 06:14:07.250531 (+ 2950us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.250586 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:07.251606 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:07.307569 (+ 55963us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2719,"thread_start_us":173,"threads_started":1}
W20260620 06:14:07.309054 30203 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:33157 (0 consecutive failures): Network error: Failed to ping master at 127.14.209.190:33157: Client connection negotiation failed: client connection to 127.14.209.190:33157: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:14:07.309737 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.309476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:49659)
0620 06:14:07.309609 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.309676 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:14:07.310426 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.310206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:51747)
0620 06:14:07.310313 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.310363 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:14:07.310674 30203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:14:07.357321 30206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:07.357537 30206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:07.357576 30206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:07.361511 30206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:07.361562 30206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:07.361617 30206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:14:07.366501 30206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:33551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=42589
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:07.367442 30206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:07.368335 30206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:07.377127 30214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:07.377293 30206 server_base.cc:1034] running on GCE node
W20260620 06:14:07.377080 30213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:07.377076 30216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:07.377830 30206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:07.378376 30206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:07.379562 30206 hybrid_clock.cc:648] HybridClock initialized: now 1781936047379535 us; error 39 us; skew 500 ppm
Jun 20 06:14:07 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936047, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:07.382072 30206 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:14:07.382839 30206 webserver.cc:460] Webserver started at http://127.14.209.130:42589/ using document root <none> and password file <none>
I20260620 06:14:07.383407 30206 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:07.383492 30206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:07.387212 30206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:07.389031 30223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.390125 30206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:14:07.390210 30206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "5cc4e35f3e8e434486496806271de862"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "07ff2d10d7349bbcebedcb70d79438a0"
server_key_iv: "2bf9bcf8a736ed85f84dbaf650bb9f16"
server_key_version: "encryptionkey@0"
I20260620 06:14:07.390725 30206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:07.399755 30206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:07.402240 30206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:07.402428 30206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:07.402921 30206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:07.403714 30206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:07.403757 30206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.403815 30206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:07.403846 30206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.412489 30206 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:33551
I20260620 06:14:07.412525 30336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:33551 every 8 connection(s)
I20260620 06:14:07.413334 30206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:14:07.415505 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30206
I20260620 06:14:07.415983 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:44709
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:33157
--webserver_port=34419
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:07.417132 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.415187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:45435)
0620 06:14:07.415714 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.416253 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":121,"threads_started":1}
W20260620 06:14:07.417614 30337 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:33157 (0 consecutive failures): Network error: Failed to ping master at 127.14.209.190:33157: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
I20260620 06:14:07.418236 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.418022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:43151)
0620 06:14:07.418123 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.418173 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 06:14:07.418987 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.418741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:50949)
0620 06:14:07.418861 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.418918 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:14:07.419198 30337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:14:07.520725 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:07.520927 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:07.520967 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:07.524847 30340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:07.524891 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:07.524986 30340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:14:07.530350 30340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:44709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=34419
--tserver_master_addrs=127.14.209.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:07.531275 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:07.532232 30340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:07.540828 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:07.540828 30346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:07.541025 30345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:07.541030 30340 server_base.cc:1034] running on GCE node
I20260620 06:14:07.541502 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:07.542081 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:07.543256 30340 hybrid_clock.cc:648] HybridClock initialized: now 1781936047543250 us; error 28 us; skew 500 ppm
Jun 20 06:14:07 dist-test-slave-zt01 krb5kdc[28896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936047, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:07.545827 30340 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:14:07.546528 30340 webserver.cc:460] Webserver started at http://127.14.209.131:34419/ using document root <none> and password file <none>
I20260620 06:14:07.546974 30340 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:07.547030 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:07.550294 30340 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:07.552148 30355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.553236 30340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:14:07.553359 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "80593aa0b16c4f098342e30662da17e2"
format_stamp: "Formatted at 2026-06-20 06:14:05 on dist-test-slave-zt01"
server_key: "f2283f4d01d178c0bfad6790d1aedf5c"
server_key_iv: "2e4eb93dd89c64ab9e9536d0a6d1237b"
server_key_version: "encryptionkey@0"
I20260620 06:14:07.553831 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:07.572520 30340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:07.575450 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:07.575632 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:07.576146 30340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:07.576882 30340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:07.576931 30340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.576969 30340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:07.576985 30340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:07.585711 30340 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:44709
I20260620 06:14:07.585740 30468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:44709 every 8 connection(s)
I20260620 06:14:07.586571 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:14:07.589872 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.588490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:44547)
0620 06:14:07.588898 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.589281 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
W20260620 06:14:07.590358 30469 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:33157 (0 consecutive failures): Network error: Failed to ping master at 127.14.209.190:33157: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
I20260620 06:14:07.590903 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.590696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:50731)
0620 06:14:07.590799 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.590847 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 06:14:07.591509 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:07.591311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:49573)
0620 06:14:07.591410 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:07.591454 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 06:14:07.591727 30469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:14:07.592455 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30340
I20260620 06:14:08.311642 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.311250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:43623)
0620 06:14:08.311497 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:08.311548 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
I20260620 06:14:08.420022 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.419670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:47729)
0620 06:14:08.419879 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:08.419937 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260620 06:14:08.592516 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.592183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:52663)
0620 06:14:08.592385 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:08.592432 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":66,"threads_started":1}
I20260620 06:14:08.944792 30476 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:08.944962 30476 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:08.945946 30476 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
Jun 20 06:14:08 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936047, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.130@KRBTEST.COM
Jun 20 06:14:08 dist-test-slave-zt01 krb5kdc[28896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936047, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.131@KRBTEST.COM
I20260620 06:14:08.959694 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.946104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:33551 (local address 127.14.209.129:47519)
0620 06:14:08.946366 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:08.946380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:08.946485 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:08.948331 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:08.948337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:08.948884 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:08.949349 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:08.949363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.950592 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.950597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:08.951582 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:08.951591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.952261 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.952264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:08.952507 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:08.953174 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:08.953186 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:08.954589 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:08.957072 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:08.957076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:08.957350 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:08.957732 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:08.957739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:08.957905 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:08.958601 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:08.958607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:08.958931 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 06:14:08.959154 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 06:14:08.959551 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":317,"spinlock_wait_cycles":29440,"thread_start_us":53,"threads_started":1}
I20260620 06:14:08.959735 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.946403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:46865 (local address 127.14.209.131:44709)
0620 06:14:08.946975 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:08.946997 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:08.947034 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:14:08.947710 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:08.947715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:08.947995 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:08.948287 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:08.949505 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.950447 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.951699 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.952144 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.952311 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:08.955189 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:08.955193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:08.955203 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:08.955227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:08.956826 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957492 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:08.957497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:08.957503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:08.957696 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957999 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:08.958002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:08.958004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:08.958468 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:08.958605 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:08.959283 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 06:14:08.959576 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":119,"threads_started":1}
I20260620 06:14:08.959694 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.946146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:44709 (local address 127.14.209.129:46865)
0620 06:14:08.946740 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:08.946754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:08.946845 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:08.948331 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:08.948337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:08.948884 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:08.949349 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:08.949362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.950590 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.950598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:08.951583 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:08.951592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.952261 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.952265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:08.952509 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:08.953117 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:08.953132 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:08.954964 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:08.956932 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:08.956938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:08.956950 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:08.957363 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:08.957781 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:08.957789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:08.957900 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:08.958601 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:08.958607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:08.958931 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 06:14:08.959154 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 06:14:08.959490 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":249,"spinlock_wait_cycles":25088,"thread_start_us":74,"threads_started":1}
I20260620 06:14:08.959731 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:08.946406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:47519 (local address 127.14.209.130:33551)
0620 06:14:08.946842 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:08.946853 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:08.946888 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:14:08.947665 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:08.947670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:08.947994 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:08.948287 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:08.949505 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.950452 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.951698 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:08.952143 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:08.952313 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:08.954824 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:08.954830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:08.954845 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:08.954880 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:08.956957 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957475 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:08.957480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:08.957484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:08.957628 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:08.957999 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:08.958002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:08.958004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:08.958468 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:08.958605 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:08.959283 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 06:14:08.959552 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":46,"threads_started":1}
W20260620 06:14:08.962023 30092 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
W20260620 06:14:08.962175 30089 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:08.962251 30089 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250fa488cb4460dac67bb7338dfb972; no voters: 5cc4e35f3e8e434486496806271de862, 80593aa0b16c4f098342e30662da17e2
I20260620 06:14:08.962450 30476 raft_consensus.cc:2726] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:14:09.312589 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:09.312317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:48247)
0620 06:14:09.312454 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:09.312509 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 06:14:09.421003 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:09.420624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:55155)
0620 06:14:09.420864 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:09.420914 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260620 06:14:09.593582 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:09.593191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:54807)
0620 06:14:09.593403 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:09.593461 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260620 06:14:10.313676 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:10.313259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:54865)
0620 06:14:10.313519 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:10.313584 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260620 06:14:10.422143 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:10.421737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:33955)
0620 06:14:10.422014 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:10.422065 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260620 06:14:10.528784 30485 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:10.528908 30485 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:10.529247 30485 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
W20260620 06:14:10.530107 30092 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
W20260620 06:14:10.530205 30089 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:10.530270 30089 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250fa488cb4460dac67bb7338dfb972; no voters: 5cc4e35f3e8e434486496806271de862, 80593aa0b16c4f098342e30662da17e2
I20260620 06:14:10.530402 30485 raft_consensus.cc:2726] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:14:10.594627 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:10.594212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:44111)
0620 06:14:10.594442 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:10.594509 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260620 06:14:11.314848 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:11.314391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:57599)
0620 06:14:11.314690 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:11.314747 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
I20260620 06:14:11.423192 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:11.422788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:44861)
0620 06:14:11.423001 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:11.423089 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260620 06:14:11.595675 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:11.595289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:43275)
0620 06:14:11.595548 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:11.595596 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":117,"threads_started":1}
I20260620 06:14:12.316270 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:12.315829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:43831)
0620 06:14:12.316119 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:12.316176 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":70,"threads_started":1}
I20260620 06:14:12.424234 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:12.423871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:39981)
0620 06:14:12.424100 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:12.424151 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":65,"threads_started":1}
I20260620 06:14:12.596750 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:12.596325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:42711)
0620 06:14:12.596613 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:12.596670 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":120,"threads_started":1}
I20260620 06:14:13.303382 30493 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:13.303536 30493 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:13.304054 30493 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
W20260620 06:14:13.304872 30092 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
W20260620 06:14:13.305004 30089 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:13.305078 30089 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250fa488cb4460dac67bb7338dfb972; no voters: 5cc4e35f3e8e434486496806271de862, 80593aa0b16c4f098342e30662da17e2
I20260620 06:14:13.305191 30493 raft_consensus.cc:2726] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:14:13.317380 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:13.317029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:53761)
0620 06:14:13.317249 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:13.317298 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260620 06:14:13.425341 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:13.424955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:50313)
0620 06:14:13.425205 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:13.425257 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260620 06:14:13.597787 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:13.597396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:55613)
0620 06:14:13.597658 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:13.597708 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260620 06:14:14.318470 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:14.318068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:39201)
0620 06:14:14.318303 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:14.318358 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260620 06:14:14.426523 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:14.426104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:34605)
0620 06:14:14.426384 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:14.426440 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":64,"threads_started":1}
I20260620 06:14:14.598798 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:14.598412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:44501)
0620 06:14:14.598672 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:14.598722 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":68,"threads_started":1}
I20260620 06:14:15.319602 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:15.319216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:38031)
0620 06:14:15.319449 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:15.319504 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260620 06:14:15.427615 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:15.427241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:48857)
0620 06:14:15.427482 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:15.427532 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
I20260620 06:14:15.599795 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:15.599437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:33525)
0620 06:14:15.599668 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:15.599717 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260620 06:14:16.320686 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:16.320293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:49691)
0620 06:14:16.320541 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:16.320595 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260620 06:14:16.428659 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:16.428268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:44923)
0620 06:14:16.428518 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:16.428572 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":67,"threads_started":1}
I20260620 06:14:16.501672 30505 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:16.501801 30505 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:16.502185 30505 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
W20260620 06:14:16.502956 30092 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
W20260620 06:14:16.503078 30089 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:16.503146 30089 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250fa488cb4460dac67bb7338dfb972; no voters: 5cc4e35f3e8e434486496806271de862, 80593aa0b16c4f098342e30662da17e2
I20260620 06:14:16.503273 30505 raft_consensus.cc:2726] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:14:16.600801 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:16.600423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:47417)
0620 06:14:16.600665 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:16.600714 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260620 06:14:17.321830 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:17.321446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:53341)
0620 06:14:17.321687 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:17.321738 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260620 06:14:17.429688 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:17.429335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:49603)
0620 06:14:17.429553 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:17.429604 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260620 06:14:17.601785 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:17.601451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:36427)
0620 06:14:17.601661 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:17.601709 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260620 06:14:18.322989 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:18.322557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:33573)
0620 06:14:18.322809 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:18.322889 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260620 06:14:18.430718 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:18.430346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:57315)
0620 06:14:18.430578 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:18.430629 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260620 06:14:18.602792 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:18.602449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:47041)
0620 06:14:18.602664 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:18.602712 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260620 06:14:19.324111 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:19.323715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:59149)
0620 06:14:19.323955 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:19.324017 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":67,"threads_started":1}
I20260620 06:14:19.431921 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:19.431499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:49265)
0620 06:14:19.431772 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:19.431823 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260620 06:14:19.603775 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:19.603406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:36083)
0620 06:14:19.603647 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:19.603697 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260620 06:14:20.214387 30516 raft_consensus.cc:482] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:20.214520 30516 raft_consensus.cc:504] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc4e35f3e8e434486496806271de862" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 33551 } } peers { permanent_uuid: "80593aa0b16c4f098342e30662da17e2" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 44709 } } peers { permanent_uuid: "9250fa488cb4460dac67bb7338dfb972" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45869 } }
I20260620 06:14:20.215003 30516 leader_election.cc:288] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551), 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709)
W20260620 06:14:20.215780 30092 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5cc4e35f3e8e434486496806271de862 (127.14.209.130:33551): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
W20260620 06:14:20.215950 30089 leader_election.cc:341] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80593aa0b16c4f098342e30662da17e2 (127.14.209.131:44709): Not found: Tablet not found: 2cb17fe4f8e641e1bf16ee7cda604aa6
I20260620 06:14:20.216019 30089 leader_election.cc:302] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9250fa488cb4460dac67bb7338dfb972; no voters: 5cc4e35f3e8e434486496806271de862, 80593aa0b16c4f098342e30662da17e2
I20260620 06:14:20.216148 30516 raft_consensus.cc:2726] T 2cb17fe4f8e641e1bf16ee7cda604aa6 P 9250fa488cb4460dac67bb7338dfb972 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:14:20.325151 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:20.324775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:38671)
0620 06:14:20.325006 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:20.325061 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260620 06:14:20.432960 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:20.432574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:34527)
0620 06:14:20.432809 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:20.432860 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260620 06:14:20.604796 30519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:20.604437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:48281)
0620 06:14:20.604662 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:20.604713 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260620 06:14:21.326236 30520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:21.325846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:48815)
0620 06:14:21.326090 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:21.326142 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260620 06:14:21.433983 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:21.433621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:38097)
0620 06:14:21.433843 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:21.433894 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260620 06:14:21.605870 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:21.605488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:41697)
0620 06:14:21.605740 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:21.605792 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260620 06:14:22.327520 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:22.327126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.129:35971)
0620 06:14:22.327371 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:22.327429 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260620 06:14:22.435213 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:22.434773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.130:47957)
0620 06:14:22.435004 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:22.435114 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260620 06:14:22.602108 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30039
I20260620 06:14:22.607306 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:22.606873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:33157 (local address 127.14.209.131:43905)
0620 06:14:22.607143 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:22.607210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260620 06:14:22.608189 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30206
I20260620 06:14:22.613794 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30340
2026-06-20T06:14:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18817 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:14:22 dist-test-slave-zt01 krb5kdc[30528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:14:22 dist-test-slave-zt01 krb5kdc[30528](info): set up 2 sockets
Jun 20 06:14:22 dist-test-slave-zt01 krb5kdc[30528](info): commencing operation
krb5kdc: starting...
W20260620 06:14:23.651233 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:14:23 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:14:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:14:23.767983 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40201
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:23.875844 30541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:23.876078 30541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:23.876116 30541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:23.879891 30541 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:14:23.879931 30541 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:14:23.879946 30541 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:14:23.879961 30541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:23.879977 30541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:23.879995 30541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:23.880010 30541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:23.880030 30541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:23.884967 30541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:23.885802 30541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:23.886660 30541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:23.895169 30549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:23.895186 30547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:23.895177 30546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:23.895238 30541 server_base.cc:1034] running on GCE node
I20260620 06:14:23.895922 30541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:23.896883 30541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:23.898066 30541 hybrid_clock.cc:648] HybridClock initialized: now 1781936063898049 us; error 29 us; skew 500 ppm
Jun 20 06:14:23 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936063, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:23.900763 30541 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:23.901573 30541 webserver.cc:460] Webserver started at http://127.14.209.190:42817/ using document root <none> and password file <none>
I20260620 06:14:23.902093 30541 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:23.902181 30541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:23.902400 30541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:23.904359 30541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "9f9ed32304c34cd29faa1e0bf957c59b"
format_stamp: "Formatted at 2026-06-20 06:14:23 on dist-test-slave-zt01"
server_key: "ef7020f978df0a194f87ddd92b27fdca"
server_key_iv: "c15398e4c7d4b78b5bbb07346f8744d0"
server_key_version: "encryptionkey@0"
I20260620 06:14:23.904906 30541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "9f9ed32304c34cd29faa1e0bf957c59b"
format_stamp: "Formatted at 2026-06-20 06:14:23 on dist-test-slave-zt01"
server_key: "ef7020f978df0a194f87ddd92b27fdca"
server_key_iv: "c15398e4c7d4b78b5bbb07346f8744d0"
server_key_version: "encryptionkey@0"
I20260620 06:14:23.908357 30541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:23.910617 30556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:23.912009 30541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:14:23.912151 30541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "9f9ed32304c34cd29faa1e0bf957c59b"
format_stamp: "Formatted at 2026-06-20 06:14:23 on dist-test-slave-zt01"
server_key: "ef7020f978df0a194f87ddd92b27fdca"
server_key_iv: "c15398e4c7d4b78b5bbb07346f8744d0"
server_key_version: "encryptionkey@0"
I20260620 06:14:23.912256 30541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:23.923575 30541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:23.926699 30541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:23.926898 30541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:23.933706 30541 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40201
I20260620 06:14:23.933987 30608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40201 every 8 connection(s)
I20260620 06:14:23.934608 30541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:23.937254 30609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:23.942229 30609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap starting.
I20260620 06:14:23.943347 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30541
I20260620 06:14:23.943471 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:14:23.943756 15174 external_mini_cluster.cc:1428] Setting key c55a0ad352f5203365adf7f3010dd7e0
I20260620 06:14:23.945017 30609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:23.945722 30609 log.cc:824] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:23.947903 30609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: No bootstrap required, opened a new log
I20260620 06:14:23.952301 30609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:23.952536 30609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:23.952618 30609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9ed32304c34cd29faa1e0bf957c59b, State: Initialized, Role: FOLLOWER
Jun 20 06:14:23 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:23.953068 30609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:23.953213 30609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:14:23.953282 30609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:14:23.953387 30609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:23.954535 30609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:23.954910 30609 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9ed32304c34cd29faa1e0bf957c59b; no voters:
I20260620 06:14:23.955253 30609 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:14:23.955706 30614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:23.955889 30614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 LEADER]: Becoming Leader. State: Replica: 9f9ed32304c34cd29faa1e0bf957c59b, State: Running, Role: LEADER
I20260620 06:14:23.956156 30614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:23.956526 30609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:23.957247 30614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } } }
I20260620 06:14:23.957376 30614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:23.957561 30614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9ed32304c34cd29faa1e0bf957c59b. Latest consensus state: current_term: 1 leader_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } } }
I20260620 06:14:23.957639 30614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:23.957986 30622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:23.959973 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:23.944840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.14.209.190:40201)
0620 06:14:23.945228 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:23.945245 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:23.945277 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:14:23.945898 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:23.945925 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:23.946308 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:23.946635 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:23.947570 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:23.948591 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:23.949815 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:23.950303 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:23.950430 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:23.953411 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:23.953417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:23.953428 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:23.953505 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:23.955854 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:23.956434 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:23.956440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:23.956446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:23.956592 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:23.956921 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:23.956925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:23.956928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:23.957482 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:23.957682 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:23.958602 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 06:14:23.958916 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260620 06:14:23.962540 30622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:23.969177 30622 catalog_manager.cc:1347] Generated new cluster ID: 5ca776d77abc47038f0f0aa50b088836
I20260620 06:14:23.969244 30622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:14:23.974097 30622 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:14:23.975274 30622 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:14:23.984675 30622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Generated new TSK 0
I20260620 06:14:23.985285 30622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:14:24.021991 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40201
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:24.143841 30635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:24.144065 30635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:24.144105 30635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:24.148187 30635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:24.148237 30635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:24.148326 30635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:14:24.153316 30635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:24.154220 30635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:24.155200 30635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:24.164340 30643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:24.164481 30635 server_base.cc:1034] running on GCE node
W20260620 06:14:24.164371 30640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:24.164362 30641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:24.164950 30635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:24.165525 30635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:24.166678 30635 hybrid_clock.cc:648] HybridClock initialized: now 1781936064166661 us; error 30 us; skew 500 ppm
Jun 20 06:14:24 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:24.169278 30635 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:14:24.170006 30635 webserver.cc:460] Webserver started at http://127.14.209.129:46731/ using document root <none> and password file <none>
I20260620 06:14:24.170480 30635 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:24.170534 30635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:24.170680 30635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:24.172544 30635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "84fdb27cdab3b4ca5dad1eff0a4a7455"
server_key_iv: "532a43d63d68fbfb89a49fc8dcbd88a6"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.173008 30635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "84fdb27cdab3b4ca5dad1eff0a4a7455"
server_key_iv: "532a43d63d68fbfb89a49fc8dcbd88a6"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.176313 30635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:14:24.178506 30650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.179713 30635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:24.179826 30635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "84fdb27cdab3b4ca5dad1eff0a4a7455"
server_key_iv: "532a43d63d68fbfb89a49fc8dcbd88a6"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.179924 30635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:24.201027 30635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:24.204293 30635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:24.204494 30635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:24.205086 30635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:24.205857 30635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:24.205902 30635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.205940 30635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:24.205966 30635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.214677 30635 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:32815
I20260620 06:14:24.214696 30763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:32815 every 8 connection(s)
I20260620 06:14:24.215585 30635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:14:24.217954 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30635
I20260620 06:14:24.218087 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:14:24.218446 15174 external_mini_cluster.cc:1428] Setting key aed79856f0999ee0778734d520605e7f
Jun 20 06:14:24 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:14:24.228902 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.217361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:60571 (local address 127.14.209.190:40201)
0620 06:14:24.217514 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.217519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.218295 (+ 776us) server_negotiation.cc:407] Connection header received
0620 06:14:24.219697 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.219700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.219750 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.219827 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:24.220878 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.221411 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.222508 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.222758 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.222872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.225315 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:24.225318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:24.225321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:24.225345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:24.226795 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.227357 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.227361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.227364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.227455 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.227795 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.227798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.227799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.227971 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:24.228076 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.228606 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.228733 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:14:24.229665 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.217717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:60571)
0620 06:14:24.218135 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.218183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.219493 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.219979 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.219986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.220305 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:24.220732 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.220744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.221536 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.221541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.222381 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.222390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.222900 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.222904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.223129 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.223711 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:24.223726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:24.225162 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:24.226911 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:24.226916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:24.226928 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:24.227256 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:24.227555 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:24.227559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:24.227561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:24.227694 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:24.228076 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:24.228082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:24.228353 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.228545 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.228834 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:14:24.230842 30764 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40201
I20260620 06:14:24.231102 30764 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:24.231650 30764 heartbeater.cc:511] Master 127.14.209.190:40201 requested a full tablet report, sending...
I20260620 06:14:24.233150 30573 ts_manager.cc:194] Registered new tserver with Master: a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815)
I20260620 06:14:24.234750 30573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:60571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:14:24.245643 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40201
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:24.350678 30769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:24.350884 30769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:24.350924 30769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:24.354701 30769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:24.354745 30769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:24.354799 30769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:14:24.359910 30769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:24.360826 30769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:24.361668 30769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:24.370172 30777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:24.370187 30774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:24.370237 30775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:24.370494 30769 server_base.cc:1034] running on GCE node
I20260620 06:14:24.370843 30769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:24.371455 30769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:24.372663 30769 hybrid_clock.cc:648] HybridClock initialized: now 1781936064372634 us; error 31 us; skew 500 ppm
Jun 20 06:14:24 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:24.375135 30769 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:14:24.375874 30769 webserver.cc:460] Webserver started at http://127.14.209.130:41489/ using document root <none> and password file <none>
I20260620 06:14:24.376385 30769 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:24.376470 30769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:24.376678 30769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:24.378455 30769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "e602c04a88cd4c72881e6b79d3f13dfe"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "aefae16997661653fa5ddfe0e8eda351"
server_key_iv: "d7519d87d17c9bac1d0c0d976431eb29"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.378963 30769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "e602c04a88cd4c72881e6b79d3f13dfe"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "aefae16997661653fa5ddfe0e8eda351"
server_key_iv: "d7519d87d17c9bac1d0c0d976431eb29"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.382143 30769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:14:24.384421 30784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.385471 30769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:14:24.385592 30769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "e602c04a88cd4c72881e6b79d3f13dfe"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "aefae16997661653fa5ddfe0e8eda351"
server_key_iv: "d7519d87d17c9bac1d0c0d976431eb29"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.385692 30769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:24.399767 30769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:24.402221 30769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:24.402403 30769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:24.402901 30769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:24.403643 30769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:24.403712 30769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.403779 30769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:24.403820 30769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.412305 30769 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:42509
I20260620 06:14:24.412334 30897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:42509 every 8 connection(s)
I20260620 06:14:24.413053 30769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:14:24.420898 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30769
I20260620 06:14:24.420993 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:14:24.421280 15174 external_mini_cluster.cc:1428] Setting key 84d0cb43bd4c3c79d077f5cac2c7897b
Jun 20 06:14:24 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:24.426102 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.414553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:38629 (local address 127.14.209.190:40201)
0620 06:14:24.414715 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.414719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.415472 (+ 753us) server_negotiation.cc:407] Connection header received
0620 06:14:24.416872 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.416875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.416931 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.417034 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:24.418078 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.418604 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.419715 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.419936 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.420047 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.422511 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:24.422514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:24.422517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:24.422547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:24.424021 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.424608 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.424613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.424615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.424695 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.425075 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.425078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.425080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.425248 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:24.425355 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.425818 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.425929 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 06:14:24.426888 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.414871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:38629)
0620 06:14:24.415297 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.415364 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.416676 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.417203 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.417209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.417524 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:24.417932 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.417944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.418729 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.418732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.419586 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.419596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.420077 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.420081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.420254 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.420793 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:24.420808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:24.422359 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:24.424218 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:24.424225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:24.424237 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:24.424504 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:24.424792 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:24.424794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:24.424796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:24.424978 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:24.425352 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:24.425358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:24.425594 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.425765 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.426038 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260620 06:14:24.427932 30898 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40201
I20260620 06:14:24.428161 30898 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:24.428580 30898 heartbeater.cc:511] Master 127.14.209.190:40201 requested a full tablet report, sending...
I20260620 06:14:24.429621 30573 ts_manager.cc:194] Registered new tserver with Master: e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509)
I20260620 06:14:24.430414 30573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:38629
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:14:24.448000 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40201
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:24.552932 30903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:24.553188 30903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:24.553263 30903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:24.557350 30903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:24.557425 30903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:24.557508 30903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:14:24.562525 30903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:24.563479 30903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:24.564370 30903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:24.573179 30911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:24.573203 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:24.573149 30908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:24.573333 30903 server_base.cc:1034] running on GCE node
I20260620 06:14:24.573766 30903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:24.574353 30903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:24.575528 30903 hybrid_clock.cc:648] HybridClock initialized: now 1781936064575511 us; error 28 us; skew 500 ppm
Jun 20 06:14:24 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:24.577975 30903 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:14:24.578771 30903 webserver.cc:460] Webserver started at http://127.14.209.131:41179/ using document root <none> and password file <none>
I20260620 06:14:24.579342 30903 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:24.579430 30903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:24.579643 30903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:14:24.581436 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "47c91e157eda41dc9d2b093403f73e7b"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "36e2b8d98e765fbbd0370fa043783d27"
server_key_iv: "edbd631bec6104e3120a93caf295ef03"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.581950 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "47c91e157eda41dc9d2b093403f73e7b"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "36e2b8d98e765fbbd0370fa043783d27"
server_key_iv: "edbd631bec6104e3120a93caf295ef03"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.585391 30903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:14:24.587556 30918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.588637 30903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:24.588763 30903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "47c91e157eda41dc9d2b093403f73e7b"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "36e2b8d98e765fbbd0370fa043783d27"
server_key_iv: "edbd631bec6104e3120a93caf295ef03"
server_key_version: "encryptionkey@0"
I20260620 06:14:24.588867 30903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:24.600819 30903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:24.603235 30903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:24.603420 30903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:24.603926 30903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:24.604643 30903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:24.604712 30903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.604777 30903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:24.604828 30903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:24.613971 30903 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:45149
I20260620 06:14:24.614001 31031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:45149 every 8 connection(s)
I20260620 06:14:24.614728 30903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:14:24.623025 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 30903
I20260620 06:14:24.623186 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
Jun 20 06:14:24 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936064, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:24.623477 15174 external_mini_cluster.cc:1428] Setting key 1cc892f3a45c7591fa1d258a6952170d
I20260620 06:14:24.627506 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.616486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:44083 (local address 127.14.209.190:40201)
0620 06:14:24.616621 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.616625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.617424 (+ 799us) server_negotiation.cc:407] Connection header received
0620 06:14:24.618547 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.618549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.618594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.618672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:24.619699 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.620235 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.621261 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.621469 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.621557 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.624019 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:24.624023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:24.624025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:24.624058 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:24.625722 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.626160 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.626162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.626164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.626231 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.626520 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.626522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.626523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.626676 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:24.626751 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.627311 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.627397 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:14:24.628219 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.616892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:44083)
0620 06:14:24.617276 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.617312 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.618325 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.618807 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.618812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.619153 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:24.619559 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.619571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.620343 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.620346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.621155 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.621163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.621587 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.621590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.621711 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.622180 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:24.622194 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:24.623833 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:24.625821 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:24.625825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:24.625835 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:24.626078 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:24.626307 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:24.626309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:24.626310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:24.626445 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:24.626802 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:24.626806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:24.627028 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.627266 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.627494 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260620 06:14:24.629247 31032 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40201
I20260620 06:14:24.629484 31032 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:24.630002 31032 heartbeater.cc:511] Master 127.14.209.190:40201 requested a full tablet report, sending...
I20260620 06:14:24.630905 30573 ts_manager.cc:194] Registered new tserver with Master: 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
I20260620 06:14:24.631508 30573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:44083
I20260620 06:14:24.636972 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:14:24.646461 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.639048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.14.209.190:40201)
0620 06:14:24.639248 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.639312 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.639341 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:14:24.639399 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.639402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.639459 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.639568 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:24.640314 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.640799 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.641769 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.642079 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.642175 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.643488 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:24.643491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:24.643493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:24.643513 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:24.644747 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.645186 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.645190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.645191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.645412 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:24.645728 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:24.645731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:24.645732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:24.645939 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:24.646051 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.646274 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.646361 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:14:24.648990 30573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:14:24.651863 30573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:14:24.665545 31046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.660390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45149 (local address 127.0.0.1:40004)
0620 06:14:24.660836 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.660848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.660918 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.661613 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.661616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.661634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:24.661720 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.661725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.662988 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.662995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.664191 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.664201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.665002 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.665006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.665096 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.665140 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.665275 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.665394 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":65,"thread_start_us":93,"threads_started":1}
I20260620 06:14:24.665553 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.660465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.14.209.130:42509)
0620 06:14:24.661087 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.661098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.661118 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:14:24.661193 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.661198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.661375 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.661557 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:24.661756 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.662440 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.664267 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.664852 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.665000 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.665135 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.665306 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.665400 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":127,"threads_started":1}
I20260620 06:14:24.665568 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.660064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:42509 (local address 127.0.0.1:55222)
0620 06:14:24.660579 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.660593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.660669 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.661538 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.661541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.661558 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:24.661656 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.661662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.662601 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.662603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.664162 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.664169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.664978 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.664982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.665107 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.665146 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.665251 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.665373 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":135,"thread_start_us":96,"threads_started":1}
I20260620 06:14:24.666149 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.660872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.14.209.131:45149)
0620 06:14:24.661170 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.661184 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.661205 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:14:24.661272 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.661278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.661441 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.661848 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:24.661979 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.662859 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.664354 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.664870 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.665008 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.665135 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.665973 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.666050 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260620 06:14:24.666810 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.659711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:32815 (local address 127.0.0.1:58302)
0620 06:14:24.660004 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.660045 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.660160 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.660793 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.660800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.660822 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:24.660967 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.660973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.662556 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.662560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.664142 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.664154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.666439 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.666443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.666542 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.666563 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.666626 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.666687 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":114,"thread_start_us":89,"threads_started":1}
I20260620 06:14:24.667160 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.659856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.14.209.129:32815)
0620 06:14:24.660297 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.660309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.660331 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:14:24.660410 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.660414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.660612 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.661130 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:24.661294 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.662417 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.665199 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.666309 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.666440 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.666788 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.666839 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.666999 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260620 06:14:24.668268 30832 tablet_service.cc:1467] Processing CreateTablet for tablet 1c27201caccf45999ef43d236b2e720b (DEFAULT_TABLE table=test-table [id=00ac9eac0a024152aa4586d632852910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:24.668309 30966 tablet_service.cc:1467] Processing CreateTablet for tablet 1c27201caccf45999ef43d236b2e720b (DEFAULT_TABLE table=test-table [id=00ac9eac0a024152aa4586d632852910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:24.669102 30832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c27201caccf45999ef43d236b2e720b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:24.669111 30966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c27201caccf45999ef43d236b2e720b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:24.673957 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 1c27201caccf45999ef43d236b2e720b (DEFAULT_TABLE table=test-table [id=00ac9eac0a024152aa4586d632852910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:24.674688 31049 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Bootstrap starting.
I20260620 06:14:24.674921 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c27201caccf45999ef43d236b2e720b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:24.675879 31050 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Bootstrap starting.
I20260620 06:14:24.676658 31049 tablet_bootstrap.cc:654] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:24.677615 31049 log.cc:824] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:24.678221 31050 tablet_bootstrap.cc:654] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:24.678999 31050 log.cc:824] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:24.679566 31049 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: No bootstrap required, opened a new log
I20260620 06:14:24.679755 31049 ts_tablet_manager.cc:1397] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:14:24.680959 31050 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: No bootstrap required, opened a new log
I20260620 06:14:24.681181 31050 ts_tablet_manager.cc:1397] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:14:24.682272 31053 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap starting.
I20260620 06:14:24.682715 31049 raft_consensus.cc:348] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.682993 31049 raft_consensus.cc:374] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:24.683112 31049 raft_consensus.cc:729] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c91e157eda41dc9d2b093403f73e7b, State: Initialized, Role: FOLLOWER
I20260620 06:14:24.683593 31049 consensus_queue.cc:260] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.684255 31032 heartbeater.cc:503] Master 127.14.209.190:40201 was elected leader, sending a full tablet report...
I20260620 06:14:24.684600 31053 tablet_bootstrap.cc:654] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:24.684638 31049 ts_tablet_manager.cc:1428] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 06:14:24.685103 31050 raft_consensus.cc:348] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.685425 31050 raft_consensus.cc:374] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:24.685489 31050 raft_consensus.cc:729] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e602c04a88cd4c72881e6b79d3f13dfe, State: Initialized, Role: FOLLOWER
I20260620 06:14:24.685705 31053 log.cc:824] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:24.685912 31050 consensus_queue.cc:260] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.686574 30898 heartbeater.cc:503] Master 127.14.209.190:40201 was elected leader, sending a full tablet report...
I20260620 06:14:24.686795 31050 ts_tablet_manager.cc:1428] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260620 06:14:24.687587 31053 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: No bootstrap required, opened a new log
I20260620 06:14:24.687755 31053 ts_tablet_manager.cc:1397] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:14:24.690865 31053 raft_consensus.cc:348] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.691037 31053 raft_consensus.cc:374] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:24.691125 31053 raft_consensus.cc:729] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a952cd6232ce4dbf9c2201461d4fee1d, State: Initialized, Role: FOLLOWER
I20260620 06:14:24.691480 31053 consensus_queue.cc:260] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.692073 30764 heartbeater.cc:503] Master 127.14.209.190:40201 was elected leader, sending a full tablet report...
I20260620 06:14:24.692291 31053 ts_tablet_manager.cc:1428] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
W20260620 06:14:24.716694 30765 tablet.cc:2369] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:14:24.808632 31056 raft_consensus.cc:482] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:24.808861 31056 raft_consensus.cc:504] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.809950 31056 leader_election.cc:288] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
I20260620 06:14:24.813644 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.810234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:32815 (local address 127.14.209.130:52587)
0620 06:14:24.810383 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.810400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.810501 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.810905 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.810907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.810922 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:24.811023 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.811028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.811788 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.811794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.812550 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.812558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.813230 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.813233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.813343 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.813360 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.813447 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.813514 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"spinlock_wait_cycles":3840}
I20260620 06:14:24.813643 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.810300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:52587 (local address 127.14.209.129:32815)
0620 06:14:24.810444 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.810449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.810464 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:14:24.810653 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.810659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.810717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.810807 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:24.811177 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.811677 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.812704 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.813107 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.813238 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.813288 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.813463 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.813525 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:14:24.814086 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.810516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45149 (local address 127.14.209.130:58515)
0620 06:14:24.810837 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.810856 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:24.810965 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:24.811253 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:24.811256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:24.811270 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:24.811400 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.811405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.812119 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.812123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.812870 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:24.812877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.813505 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.813509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:24.813613 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.813635 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:14:24.813883 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 06:14:24.813952 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260620 06:14:24.814082 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.810652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:58515 (local address 127.14.209.131:45149)
0620 06:14:24.810795 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.810799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.810935 (+ 136us) server_negotiation.cc:407] Connection header received
0620 06:14:24.811110 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.811112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.811159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.811243 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:24.811508 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.811997 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.812991 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.813388 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.813512 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.813558 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.813904 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.813971 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 06:14:24.814307 30718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27201caccf45999ef43d236b2e720b" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" is_pre_election: true
I20260620 06:14:24.814616 30718 raft_consensus.cc:2454] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 0.
I20260620 06:14:24.814682 30986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27201caccf45999ef43d236b2e720b" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c91e157eda41dc9d2b093403f73e7b" is_pre_election: true
I20260620 06:14:24.814947 30986 raft_consensus.cc:2454] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 0.
I20260620 06:14:24.815131 30787 leader_election.cc:302] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d, e602c04a88cd4c72881e6b79d3f13dfe; no voters:
I20260620 06:14:24.815439 31056 raft_consensus.cc:2781] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:14:24.815533 31056 raft_consensus.cc:482] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:14:24.815593 31056 raft_consensus.cc:3037] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:24.816648 31056 raft_consensus.cc:504] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.816955 31056 leader_election.cc:288] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 election: Requested vote from peers a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
I20260620 06:14:24.817633 30718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27201caccf45999ef43d236b2e720b" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
I20260620 06:14:24.817770 30718 raft_consensus.cc:3037] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:24.817823 30986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c27201caccf45999ef43d236b2e720b" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c91e157eda41dc9d2b093403f73e7b"
I20260620 06:14:24.818071 30986 raft_consensus.cc:3037] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:24.818895 30718 raft_consensus.cc:2454] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 1.
I20260620 06:14:24.819217 30787 leader_election.cc:302] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d, e602c04a88cd4c72881e6b79d3f13dfe; no voters:
I20260620 06:14:24.819398 31056 raft_consensus.cc:2781] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:24.819484 30986 raft_consensus.cc:2454] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 1.
I20260620 06:14:24.819626 31056 raft_consensus.cc:686] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 1 LEADER]: Becoming Leader. State: Replica: e602c04a88cd4c72881e6b79d3f13dfe, State: Running, Role: LEADER
I20260620 06:14:24.820113 31056 consensus_queue.cc:237] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:24.823232 30572 catalog_manager.cc:5495] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe reported cstate change: term changed from 0 to 1, leader changed from <none> to e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130). New cstate: current_term: 1 leader_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } health_report { overall_health: HEALTHY } } }
I20260620 06:14:24.860594 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.856751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.14.209.130:42509)
0620 06:14:24.856920 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:24.856924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:24.856939 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:14:24.857084 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:24.857087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:24.857143 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:24.857236 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:14:24.857547 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.858370 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.859266 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:24.859513 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:24.859592 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:24.859881 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:14:24.860262 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:14:24.860352 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:24.860440 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 06:14:24.860492 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 06:14:24.865707 31033 tablet.cc:2369] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:14:24.866159 30718 raft_consensus.cc:1261] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Refusing update from remote peer e602c04a88cd4c72881e6b79d3f13dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:24.866156 30986 raft_consensus.cc:1261] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Refusing update from remote peer e602c04a88cd4c72881e6b79d3f13dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:24.866845 31059 consensus_queue.cc:1035] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:24.867008 31056 consensus_queue.cc:1035] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:24.874753 31064 mvcc.cc:205] Tried to move back new op lower bound from 7298810121684639744 to 7298810121505636352. Current Snapshot: MvccSnapshot[applied={T|T < 7298810121684639744}]
I20260620 06:14:24.875133 31063 mvcc.cc:205] Tried to move back new op lower bound from 7298810121684639744 to 7298810121505636352. Current Snapshot: MvccSnapshot[applied={T|T < 7298810121684639744}]
I20260620 06:14:24.881857 30573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:24.882021 30573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:24.884429 30573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Sending DeleteTablet for 3 replicas of tablet 1c27201caccf45999ef43d236b2e720b
I20260620 06:14:24.885120 30966 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27201caccf45999ef43d236b2e720b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:40004
I20260620 06:14:24.885124 30832 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27201caccf45999ef43d236b2e720b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:55222
I20260620 06:14:24.885205 30698 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27201caccf45999ef43d236b2e720b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:58302
I20260620 06:14:24.885535 31072 tablet_replica.cc:330] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: stopping tablet replica
I20260620 06:14:24.885562 31073 tablet_replica.cc:330] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: stopping tablet replica
I20260620 06:14:24.885758 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30635
I20260620 06:14:24.885824 31072 raft_consensus.cc:2229] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:14:24.885845 31073 raft_consensus.cc:2229] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:24.886127 31073 raft_consensus.cc:2258] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:24.886163 31072 raft_consensus.cc:2258] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:24.887627 31072 ts_tablet_manager.cc:1905] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:24.887627 31073 ts_tablet_manager.cc:1905] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:24.890338 31072 ts_tablet_manager.cc:1918] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:24.890338 31073 ts_tablet_manager.cc:1918] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:24.890437 31072 log.cc:1196] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/1c27201caccf45999ef43d236b2e720b
I20260620 06:14:24.890444 31073 log.cc:1196] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/1c27201caccf45999ef43d236b2e720b
I20260620 06:14:24.890751 31072 ts_tablet_manager.cc:1939] T 1c27201caccf45999ef43d236b2e720b P e602c04a88cd4c72881e6b79d3f13dfe: Deleting consensus metadata
I20260620 06:14:24.890751 31073 ts_tablet_manager.cc:1939] T 1c27201caccf45999ef43d236b2e720b P 47c91e157eda41dc9d2b093403f73e7b: Deleting consensus metadata
I20260620 06:14:24.891831 30557 catalog_manager.cc:4841] TS e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): tablet 1c27201caccf45999ef43d236b2e720b (table test-table [id=00ac9eac0a024152aa4586d632852910]) successfully deleted
I20260620 06:14:24.891929 30559 catalog_manager.cc:4841] TS 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): tablet 1c27201caccf45999ef43d236b2e720b (table test-table [id=00ac9eac0a024152aa4586d632852910]) successfully deleted
I20260620 06:14:24.896785 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30769
W20260620 06:14:24.896793 30559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:32815 (error 108)
I20260620 06:14:24.898443 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:24.898179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:32815 (local address 127.0.0.1:58306)
0620 06:14:24.898303 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:24.898371 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:14:24.898618 30559 catalog_manager.cc:4584] TS a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c27201caccf45999ef43d236b2e720b: Network error: Client connection negotiation failed: client connection to 127.14.209.129:32815: connect: Connection refused (error 111)
I20260620 06:14:24.902030 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30903
I20260620 06:14:24.907413 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 30541
I20260620 06:14:44.913772 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40201
--webserver_interface=127.14.209.190
--webserver_port=42817
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:14:45.018898 31076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:45.019204 31076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:45.019248 31076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:45.023025 31076 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:14:45.023082 31076 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:14:45.023100 31076 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:14:45.023113 31076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:14:45.023131 31076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:45.023149 31076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:45.023167 31076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:14:45.023183 31076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:14:45.028149 31076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=42817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:45.029000 31076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:45.029884 31076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:14:45.042193 31076 server_base.cc:1034] running on GCE node
W20260620 06:14:45.042101 31083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.042104 31085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.042135 31082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:45.042925 31076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:45.044015 31076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:45.045203 31076 hybrid_clock.cc:648] HybridClock initialized: now 1781936085045192 us; error 25 us; skew 500 ppm
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:45.051901 31076 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:14:45.052752 31076 webserver.cc:460] Webserver started at http://127.14.209.190:42817/ using document root <none> and password file <none>
I20260620 06:14:45.053321 31076 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:45.053424 31076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:45.057112 31076 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:14:45.059270 31092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.060473 31076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:14:45.060624 31076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "9f9ed32304c34cd29faa1e0bf957c59b"
format_stamp: "Formatted at 2026-06-20 06:14:23 on dist-test-slave-zt01"
server_key: "ef7020f978df0a194f87ddd92b27fdca"
server_key_iv: "c15398e4c7d4b78b5bbb07346f8744d0"
server_key_version: "encryptionkey@0"
I20260620 06:14:45.061226 31076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:45.082574 31076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:45.085440 31076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:45.085649 31076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:45.092693 31076 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40201
I20260620 06:14:45.093048 31144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40201 every 8 connection(s)
I20260620 06:14:45.093739 31076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:14:45.097922 31145 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:14:45.099519 31145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap starting.
I20260620 06:14:45.100121 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31076
I20260620 06:14:45.100540 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:32815
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:40201
--webserver_port=46731
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:45.116159 31145 log.cc:824] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:45.124300 31145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:14:45.124634 31145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap complete.
I20260620 06:14:45.128595 31145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:45.128995 31145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9ed32304c34cd29faa1e0bf957c59b, State: Initialized, Role: FOLLOWER
I20260620 06:14:45.129442 31145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:45.129580 31145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:14:45.129669 31145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:14:45.129787 31145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:14:45.131604 31145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:45.131920 31145 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9ed32304c34cd29faa1e0bf957c59b; no voters:
I20260620 06:14:45.132239 31145 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:14:45.132315 31149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:14:45.132542 31149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 LEADER]: Becoming Leader. State: Replica: 9f9ed32304c34cd29faa1e0bf957c59b, State: Running, Role: LEADER
I20260620 06:14:45.132869 31149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:14:45.133436 31145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:14:45.134282 31151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } } }
I20260620 06:14:45.134421 31151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:45.134745 31158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:14:45.134279 31150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9ed32304c34cd29faa1e0bf957c59b. Latest consensus state: current_term: 2 leader_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } } }
I20260620 06:14:45.134871 31150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: This master's current role is: LEADER
I20260620 06:14:45.139325 31158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:14:45.139952 31158 catalog_manager.cc:1259] Loaded cluster ID: 5ca776d77abc47038f0f0aa50b088836
I20260620 06:14:45.140015 31158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:14:45.142418 31158 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:14:45.143118 31158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Loaded TSK: 0
I20260620 06:14:45.156279 31158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Generated new TSK 1
I20260620 06:14:45.156513 31158 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:14:45.210896 31147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:45.211184 31147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:45.211267 31147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:45.215196 31147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:45.215266 31147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:45.215363 31147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:14:45.220407 31147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:32815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=46731
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:45.221370 31147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:45.222352 31147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:45.232249 31172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.232235 31173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.232348 31175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:45.232754 31147 server_base.cc:1034] running on GCE node
I20260620 06:14:45.233160 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:45.233893 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:45.235097 31147 hybrid_clock.cc:648] HybridClock initialized: now 1781936085235066 us; error 40 us; skew 500 ppm
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:45.238435 31147 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:14:45.239383 31147 webserver.cc:460] Webserver started at http://127.14.209.129:46731/ using document root <none> and password file <none>
I20260620 06:14:45.239949 31147 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:45.240054 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:45.243700 31147 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:14:45.245759 31182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.246966 31147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:45.247138 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "84fdb27cdab3b4ca5dad1eff0a4a7455"
server_key_iv: "532a43d63d68fbfb89a49fc8dcbd88a6"
server_key_version: "encryptionkey@0"
I20260620 06:14:45.247684 31147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:45.266834 31147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:45.269691 31147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:45.269882 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:45.270402 31147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:45.271478 31189 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:14:45.275126 31147 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:14:45.275187 31147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 06:14:45.275254 31147 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:14:45.276858 31147 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:14:45.276914 31147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:14:45.277154 31189 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap starting.
I20260620 06:14:45.287555 31147 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:32815
I20260620 06:14:45.287575 31296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:32815 every 8 connection(s)
I20260620 06:14:45.288903 31147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:14:45.296768 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31147
I20260620 06:14:45.296931 31189 log.cc:824] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:45.297278 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:42509
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:40201
--webserver_port=41489
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:45.304268 31189 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:14:45.304677 31189 tablet_bootstrap.cc:492] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap complete.
I20260620 06:14:45.305946 31189 ts_tablet_manager.cc:1397] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.009s
I20260620 06:14:45.306950 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.290890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:51163)
0620 06:14:45.291466 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.291505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.292308 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.293329 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.293337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.293751 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:45.294168 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.294181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.295504 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.295507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.296324 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.296333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.297150 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.297154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.297350 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.298849 (+ 1499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:45.298874 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:45.300753 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:45.303499 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.303504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.303516 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.303844 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.304277 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.304283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.304288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.304464 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.305230 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:45.305238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:45.305522 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.305717 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.306094 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":209,"threads_started":1}
I20260620 06:14:45.307125 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.290870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:51163 (local address 127.14.209.190:40201)
0620 06:14:45.291326 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.291344 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.291644 (+ 300us) server_negotiation.cc:407] Connection header received
0620 06:14:45.292687 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.292697 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.292959 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.293278 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:45.294338 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.295334 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.296449 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.296975 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.297162 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.300960 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:45.300970 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:45.300988 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:45.301027 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:45.303355 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.303972 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.303980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.303989 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.304160 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.304577 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.304583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.304587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.305100 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:45.305251 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.305871 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.306202 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":95,"threads_started":1}
I20260620 06:14:45.308748 31189 raft_consensus.cc:348] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:45.309159 31189 raft_consensus.cc:729] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a952cd6232ce4dbf9c2201461d4fee1d, State: Initialized, Role: FOLLOWER
I20260620 06:14:45.309660 31189 consensus_queue.cc:260] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } }
I20260620 06:14:45.309998 31297 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40201
I20260620 06:14:45.310216 31297 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:45.310698 31189 ts_tablet_manager.cc:1428] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 06:14:45.310739 31297 heartbeater.cc:511] Master 127.14.209.190:40201 requested a full tablet report, sending...
I20260620 06:14:45.313135 31109 ts_manager.cc:194] Registered new tserver with Master: a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815)
I20260620 06:14:45.316946 31109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:51163
I20260620 06:14:45.319254 31297 heartbeater.cc:503] Master 127.14.209.190:40201 was elected leader, sending a full tablet report...
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.129@KRBTEST.COM
I20260620 06:14:45.326921 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.316328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:32815 (local address 127.0.0.1:57738)
0620 06:14:45.316882 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.316930 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.317105 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.317566 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.317571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.318019 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:45.318143 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.318151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.319589 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.319592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.320238 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.320248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.320913 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.320917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.321031 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.321540 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:45.321554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:45.323032 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:45.325280 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.325288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.325293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.325586 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.325883 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.325885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.325886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.325962 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.326423 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:45.326427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:45.326555 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.326666 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.326803 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":96,"threads_started":1}
I20260620 06:14:45.327091 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.316381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.14.209.129:32815)
0620 06:14:45.317042 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.317054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.317139 (+ 85us) server_negotiation.cc:407] Connection header received
0620 06:14:45.317262 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.317265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.317419 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.317562 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:45.318277 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.319467 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.320362 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.320790 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.320899 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.323215 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:45.323224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:45.323236 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:45.323260 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:45.325148 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.325701 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.325706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.325708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.325793 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.326053 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.326056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.326057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.326317 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:45.326437 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.326769 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.326917 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":102,"threads_started":1}
I20260620 06:14:45.327939 31231 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c27201caccf45999ef43d236b2e720b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:57738
I20260620 06:14:45.328737 31306 tablet_replica.cc:330] stopping tablet replica
I20260620 06:14:45.329021 31306 raft_consensus.cc:2229] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:45.329260 31306 raft_consensus.cc:2258] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:45.330235 31306 ts_tablet_manager.cc:1905] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:45.334388 31306 ts_tablet_manager.cc:1918] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:45.334479 31306 log.cc:1196] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/1c27201caccf45999ef43d236b2e720b
I20260620 06:14:45.334735 31306 ts_tablet_manager.cc:1939] T 1c27201caccf45999ef43d236b2e720b P a952cd6232ce4dbf9c2201461d4fee1d: Deleting consensus metadata
I20260620 06:14:45.336862 31095 catalog_manager.cc:4841] TS a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815): tablet 1c27201caccf45999ef43d236b2e720b (table test-table [id=00ac9eac0a024152aa4586d632852910]) successfully deleted
W20260620 06:14:45.415500 31302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:45.415715 31302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:45.415755 31302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:45.419552 31302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:45.419596 31302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:45.419651 31302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:14:45.424482 31302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:42509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=41489
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:45.425300 31302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:45.426149 31302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:45.435431 31311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.435408 31312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.435427 31314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:45.435742 31302 server_base.cc:1034] running on GCE node
I20260620 06:14:45.436189 31302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:45.436754 31302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:45.437906 31302 hybrid_clock.cc:648] HybridClock initialized: now 1781936085437877 us; error 41 us; skew 500 ppm
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:45.440680 31302 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:14:45.441430 31302 webserver.cc:460] Webserver started at http://127.14.209.130:41489/ using document root <none> and password file <none>
I20260620 06:14:45.441932 31302 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:45.442024 31302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:45.445467 31302 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:45.447497 31321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.448606 31302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:14:45.448717 31302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "e602c04a88cd4c72881e6b79d3f13dfe"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "aefae16997661653fa5ddfe0e8eda351"
server_key_iv: "d7519d87d17c9bac1d0c0d976431eb29"
server_key_version: "encryptionkey@0"
I20260620 06:14:45.449191 31302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:45.473685 31302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:45.476675 31302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:45.476899 31302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:45.477499 31302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:45.478434 31302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:45.478508 31302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.478567 31302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:45.478618 31302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.487847 31302 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:42509
I20260620 06:14:45.487871 31434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:42509 every 8 connection(s)
I20260620 06:14:45.488817 31302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:45.498724 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31302
I20260620 06:14:45.499204 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:45149
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:40201
--webserver_port=41179
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:14:45.501842 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.490641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:40125 (local address 127.14.209.190:40201)
0620 06:14:45.490786 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.490789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.491561 (+ 772us) server_negotiation.cc:407] Connection header received
0620 06:14:45.492787 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.492790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.492849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.492951 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:45.494008 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.494535 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.495762 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.495999 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.496117 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.498311 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:45.498315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:45.498319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:45.498348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:45.499904 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.500383 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.500387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.500389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.500471 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.500808 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.500812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.500813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.500988 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:45.501094 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.501539 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.501675 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:14:45.502569 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.490949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:40125)
0620 06:14:45.491417 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.491454 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.492582 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.493106 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.493117 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.493432 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:45.493869 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.493881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.494679 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.494687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.495637 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.495646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.496156 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.496160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.496317 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.496811 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:45.496826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:45.498152 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:45.500020 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.500025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.500037 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.500281 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.500569 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.500571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.500573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.500706 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.501093 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:45.501099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:45.501323 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.501498 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.501782 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
I20260620 06:14:45.503991 31435 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40201
I20260620 06:14:45.504217 31435 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:45.504660 31435 heartbeater.cc:511] Master 127.14.209.190:40201 requested a full tablet report, sending...
I20260620 06:14:45.505743 31109 ts_manager.cc:194] Registered new tserver with Master: e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509)
I20260620 06:14:45.506596 31109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:40125
W20260620 06:14:45.605114 31438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:14:45.605329 31438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:14:45.605368 31438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:14:45.609257 31438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:14:45.609311 31438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:14:45.609365 31438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:14:45.614302 31438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:45149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=41179
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:14:45.615262 31438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:14:45.616168 31438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:14:45.625113 31443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.625118 31444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:14:45.625190 31446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:14:45.625370 31438 server_base.cc:1034] running on GCE node
I20260620 06:14:45.625808 31438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:14:45.626441 31438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:14:45.627626 31438 hybrid_clock.cc:648] HybridClock initialized: now 1781936085627595 us; error 44 us; skew 500 ppm
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:14:45.630319 31438 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:14:45.631211 31438 webserver.cc:460] Webserver started at http://127.14.209.131:41179/ using document root <none> and password file <none>
I20260620 06:14:45.631727 31438 fs_manager.cc:359] Metadata directory not provided
I20260620 06:14:45.631825 31438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:14:45.635670 31438 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 06:14:45.637841 31453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.639086 31438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:14:45.639213 31438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "47c91e157eda41dc9d2b093403f73e7b"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "36e2b8d98e765fbbd0370fa043783d27"
server_key_iv: "edbd631bec6104e3120a93caf295ef03"
server_key_version: "encryptionkey@0"
I20260620 06:14:45.639753 31438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:14:45.665972 31438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:14:45.669044 31438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:14:45.669237 31438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:14:45.669821 31438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:14:45.670697 31438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:14:45.670769 31438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.670836 31438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:14:45.670871 31438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:14:45.680646 31438 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:45149
I20260620 06:14:45.680678 31566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:45149 every 8 connection(s)
I20260620 06:14:45.681596 31438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:14:45.685652 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31438
Jun 20 06:14:45 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:14:45.695214 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.686322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.14.209.190:40201)
0620 06:14:45.686962 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.686967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.686979 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:14:45.687014 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.687016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.687084 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.687170 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:45.687862 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.688483 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.689675 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.690110 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.690212 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.691901 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:45.691908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:45.691914 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:45.691946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:45.693550 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.693974 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.693980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.693986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.694061 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.694331 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.694335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.694337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.694508 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:45.694739 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.694872 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.694992 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":572,"thread_start_us":86,"threads_started":1}
I20260620 06:14:45.696923 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.683487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:60727 (local address 127.14.209.190:40201)
0620 06:14:45.683683 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.683687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.684507 (+ 820us) server_negotiation.cc:407] Connection header received
0620 06:14:45.685782 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.685786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.685844 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.685950 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:45.687490 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.688150 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.689675 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.690110 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.690237 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.693215 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:45.693218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:45.693220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:45.693242 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:45.694668 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.695220 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.695227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.695231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.695315 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.695738 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.695745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.695750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.695948 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:45.696049 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.696684 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.696771 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":74,"spinlock_wait_cycles":41216}
I20260620 06:14:45.697706 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.683852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:60727)
0620 06:14:45.684280 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.684327 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.685570 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.686122 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.686128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.686639 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:14:45.687339 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.687356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.688273 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.688276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.689547 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.689559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.690257 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.690261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.690447 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.691018 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:14:45.691039 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:14:45.693016 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:14:45.694788 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.694794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.694807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.695116 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.695403 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:14:45.695406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:14:45.695408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:14:45.695599 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:14:45.696081 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:14:45.696088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:14:45.696380 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.696619 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.696944 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260620 06:14:45.699134 31567 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40201
I20260620 06:14:45.699352 31567 heartbeater.cc:464] Registering TS with master...
I20260620 06:14:45.699786 31567 heartbeater.cc:511] Master 127.14.209.190:40201 requested a full tablet report, sending...
I20260620 06:14:45.700747 31109 ts_manager.cc:194] Registered new tserver with Master: 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
I20260620 06:14:45.701261 31109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:60727
I20260620 06:14:45.709227 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:14:45.713594 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.709885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.14.209.190:40201)
0620 06:14:45.710011 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.710014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.710216 (+ 202us) server_negotiation.cc:407] Connection header received
0620 06:14:45.710336 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.710339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.710426 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.710514 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:14:45.710778 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.711336 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.712376 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.712816 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.712921 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.713164 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:14:45.713283 (+ 119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 06:14:45.713463 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51772: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 06:14:45.713676 31299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51772: token expired
W20260620 06:14:45.713990 31037 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.209.190:40201)
I20260620 06:14:45.721462 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.714370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.14.209.190:40201)
0620 06:14:45.714487 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.714490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.714501 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:14:45.714570 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.714572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.714603 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.714665 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:14:45.715413 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.715898 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.716734 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.716948 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.717049 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.718656 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:14:45.718659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:14:45.718661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:14:45.718681 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:14:45.719992 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.720398 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.720401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.720402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.720478 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:14:45.720730 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:14:45.720732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:14:45.720733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:14:45.720883 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:14:45.720980 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.721266 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.721362 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:14:45.723799 31109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:14:45.726852 31109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:14:45.735231 31231 tablet_service.cc:1467] Processing CreateTablet for tablet eb785a60561b478393076e545a2ac8dd (DEFAULT_TABLE table=test-table [id=e8fdf95de66946c299ad457b888e10e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:45.735888 31231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb785a60561b478393076e545a2ac8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:45.738776 31189 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap starting.
I20260620 06:14:45.739476 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.734243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45149 (local address 127.0.0.1:34958)
0620 06:14:45.734441 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.734455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.734547 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.735495 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.735498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.735515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:45.735593 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.735598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.736939 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.736942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.737849 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.737860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.738995 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.738998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.739136 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.739181 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.739257 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.739339 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260620 06:14:45.739524 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.734396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.14.209.131:45149)
0620 06:14:45.734847 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.734859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.734878 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:14:45.734943 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.734948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.735160 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.735290 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:45.735733 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.736791 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.737989 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.738881 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.738993 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.739139 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.739295 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.739388 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":154,"threads_started":1}
I20260620 06:14:45.739764 31189 tablet_bootstrap.cc:654] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:45.739786 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.733826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:42509 (local address 127.0.0.1:47426)
0620 06:14:45.733995 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.734011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.734083 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.734871 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.734874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.734893 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:45.734991 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.734997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.736770 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.736773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.737951 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.737962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.739352 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.739355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.739445 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.739463 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.739535 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.739627 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 06:14:45.740564 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.734019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.14.209.130:42509)
0620 06:14:45.734343 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.734356 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.734380 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:14:45.734462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.734468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.734695 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.735176 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:45.735368 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.736649 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.738096 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.739000 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.739150 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.739273 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.740274 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.740396 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260620 06:14:45.741957 31189 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: No bootstrap required, opened a new log
I20260620 06:14:45.742040 31189 ts_tablet_manager.cc:1397] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:14:45.742457 31189 raft_consensus.cc:348] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.742586 31189 raft_consensus.cc:374] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:45.742627 31189 raft_consensus.cc:729] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a952cd6232ce4dbf9c2201461d4fee1d, State: Initialized, Role: FOLLOWER
I20260620 06:14:45.742748 31189 consensus_queue.cc:260] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.743588 31369 tablet_service.cc:1467] Processing CreateTablet for tablet eb785a60561b478393076e545a2ac8dd (DEFAULT_TABLE table=test-table [id=e8fdf95de66946c299ad457b888e10e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:45.743614 31501 tablet_service.cc:1467] Processing CreateTablet for tablet eb785a60561b478393076e545a2ac8dd (DEFAULT_TABLE table=test-table [id=e8fdf95de66946c299ad457b888e10e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:14:45.744315 31189 ts_tablet_manager.cc:1428] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:14:45.744637 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb785a60561b478393076e545a2ac8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:45.744655 31501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb785a60561b478393076e545a2ac8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:14:45.751472 31580 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Bootstrap starting.
I20260620 06:14:45.751502 31581 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Bootstrap starting.
I20260620 06:14:45.754230 31581 tablet_bootstrap.cc:654] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:45.754236 31580 tablet_bootstrap.cc:654] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Neither blocks nor log segments found. Creating new log.
I20260620 06:14:45.754921 31580 log.cc:824] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:45.755048 31581 log.cc:824] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Log is configured to *not* fsync() on all Append() calls
I20260620 06:14:45.756675 31581 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: No bootstrap required, opened a new log
I20260620 06:14:45.756676 31580 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: No bootstrap required, opened a new log
I20260620 06:14:45.756862 31581 ts_tablet_manager.cc:1397] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:14:45.756868 31580 ts_tablet_manager.cc:1397] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:14:45.759893 31581 raft_consensus.cc:348] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.759892 31580 raft_consensus.cc:348] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.760102 31581 raft_consensus.cc:374] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:45.760169 31581 raft_consensus.cc:729] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e602c04a88cd4c72881e6b79d3f13dfe, State: Initialized, Role: FOLLOWER
I20260620 06:14:45.760176 31580 raft_consensus.cc:374] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:14:45.760259 31580 raft_consensus.cc:729] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c91e157eda41dc9d2b093403f73e7b, State: Initialized, Role: FOLLOWER
I20260620 06:14:45.760617 31581 consensus_queue.cc:260] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.760664 31580 consensus_queue.cc:260] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.761193 31435 heartbeater.cc:503] Master 127.14.209.190:40201 was elected leader, sending a full tablet report...
I20260620 06:14:45.761245 31567 heartbeater.cc:503] Master 127.14.209.190:40201 was elected leader, sending a full tablet report...
I20260620 06:14:45.761536 31581 ts_tablet_manager.cc:1428] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:14:45.761555 31580 ts_tablet_manager.cc:1428] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 06:14:45.790169 31298 tablet.cc:2369] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:14:45.932567 31568 tablet.cc:2369] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:14:45.948132 31584 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:14:45.948307 31584 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.949400 31584 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149), a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815)
I20260620 06:14:45.954010 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.949633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45149 (local address 127.14.209.130:48813)
0620 06:14:45.949761 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.949778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.949872 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.950190 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.950193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.950220 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:45.950351 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.950357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.951412 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.951415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.952668 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.952678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.953538 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.953541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.953628 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.953648 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.953763 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.953848 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:14:45.954031 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.949717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:48813 (local address 127.14.209.131:45149)
0620 06:14:45.949850 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.949853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.949888 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:14:45.949955 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.949958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.950009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.950127 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:45.950462 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.951308 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.952779 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.953392 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.953510 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.953561 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.953822 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.953899 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:14:45.954176 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.949923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:32815 (local address 127.14.209.130:39517)
0620 06:14:45.950189 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:45.950202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:14:45.950277 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:14:45.950655 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:14:45.950657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:14:45.950679 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:14:45.950772 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.950776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.951663 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.951666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.952814 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:14:45.952820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.953773 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.953776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:14:45.953873 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.953893 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:14:45.953952 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 06:14:45.954008 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260620 06:14:45.954527 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.950061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:39517 (local address 127.14.209.129:32815)
0620 06:14:45.950316 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.950320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.950330 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:14:45.950373 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.950375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.950477 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.950631 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:14:45.950880 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.951585 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.952914 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.953641 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.953807 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.953946 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.953980 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.954047 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
I20260620 06:14:45.954718 31521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb785a60561b478393076e545a2ac8dd" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c91e157eda41dc9d2b093403f73e7b" is_pre_election: true
I20260620 06:14:45.954981 31251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb785a60561b478393076e545a2ac8dd" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" is_pre_election: true
I20260620 06:14:45.955042 31521 raft_consensus.cc:2454] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 0.
I20260620 06:14:45.955287 31251 raft_consensus.cc:2454] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 0.
I20260620 06:14:45.955582 31324 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe; no voters:
I20260620 06:14:45.955853 31584 raft_consensus.cc:2781] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:14:45.955945 31584 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:14:45.955991 31584 raft_consensus.cc:3037] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:45.957010 31584 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.957327 31584 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 election: Requested vote from peers 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149), a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815)
I20260620 06:14:45.957646 31521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb785a60561b478393076e545a2ac8dd" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c91e157eda41dc9d2b093403f73e7b"
I20260620 06:14:45.957703 31251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb785a60561b478393076e545a2ac8dd" candidate_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
I20260620 06:14:45.957762 31521 raft_consensus.cc:3037] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:45.957834 31251 raft_consensus.cc:3037] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:14:45.958737 31521 raft_consensus.cc:2454] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 1.
I20260620 06:14:45.958822 31251 raft_consensus.cc:2454] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e602c04a88cd4c72881e6b79d3f13dfe in term 1.
I20260620 06:14:45.959028 31324 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe; no voters:
I20260620 06:14:45.959193 31584 raft_consensus.cc:2781] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:14:45.959419 31584 raft_consensus.cc:686] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 1 LEADER]: Becoming Leader. State: Replica: e602c04a88cd4c72881e6b79d3f13dfe, State: Running, Role: LEADER
I20260620 06:14:45.959745 31584 consensus_queue.cc:237] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:14:45.963165 31109 catalog_manager.cc:5495] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe reported cstate change: term changed from 0 to 1, leader changed from <none> to e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130). New cstate: current_term: 1 leader_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } health_report { overall_health: UNKNOWN } } }
I20260620 06:14:45.974637 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:45.971099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.14.209.130:42509)
0620 06:14:45.971202 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:14:45.971206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:14:45.971298 (+ 92us) server_negotiation.cc:407] Connection header received
0620 06:14:45.971480 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:14:45.971482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:14:45.971539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:14:45.971650 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:14:45.971880 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.972414 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.973253 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:14:45.973466 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:14:45.973581 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:14:45.973867 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:14:45.974232 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:14:45.974360 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:14:45.974425 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 06:14:45.974476 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:14:45.980415 31251 raft_consensus.cc:1261] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Refusing update from remote peer e602c04a88cd4c72881e6b79d3f13dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:45.980419 31521 raft_consensus.cc:1261] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Refusing update from remote peer e602c04a88cd4c72881e6b79d3f13dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:14:45.981060 31584 consensus_queue.cc:1035] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:45.981293 31588 consensus_queue.cc:1035] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:14:45.986667 31590 mvcc.cc:205] Tried to move back new op lower bound from 7298810208169275392 to 7298810208093450240. Current Snapshot: MvccSnapshot[applied={T|T < 7298810208169275392}]
I20260620 06:14:45.986824 31591 mvcc.cc:205] Tried to move back new op lower bound from 7298810208169275392 to 7298810208093450240. Current Snapshot: MvccSnapshot[applied={T|T < 7298810208169275392}]
I20260620 06:14:45.996130 31109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:45.996256 31109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:14:45.998618 31109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Sending DeleteTablet for 3 replicas of tablet eb785a60561b478393076e545a2ac8dd
I20260620 06:14:45.999270 31369 tablet_service.cc:1514] Processing DeleteTablet for tablet eb785a60561b478393076e545a2ac8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:45 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:47426
I20260620 06:14:45.999330 31231 tablet_service.cc:1514] Processing DeleteTablet for tablet eb785a60561b478393076e545a2ac8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:45 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:57738
I20260620 06:14:45.999449 31501 tablet_service.cc:1514] Processing DeleteTablet for tablet eb785a60561b478393076e545a2ac8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:14:45 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:34958
I20260620 06:14:45.999732 31600 tablet_replica.cc:330] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: stopping tablet replica
I20260620 06:14:45.999868 31602 tablet_replica.cc:330] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: stopping tablet replica
I20260620 06:14:46.000053 31600 raft_consensus.cc:2229] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:14:46.000125 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31147
I20260620 06:14:46.000139 31602 raft_consensus.cc:2229] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:14:46.000423 31602 raft_consensus.cc:2258] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:46.000442 31600 raft_consensus.cc:2258] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:14:46.001907 31600 ts_tablet_manager.cc:1905] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:46.003901 31602 ts_tablet_manager.cc:1905] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:14:46.004642 31600 ts_tablet_manager.cc:1918] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:46.004733 31600 log.cc:1196] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/eb785a60561b478393076e545a2ac8dd
I20260620 06:14:46.005082 31600 ts_tablet_manager.cc:1939] T eb785a60561b478393076e545a2ac8dd P e602c04a88cd4c72881e6b79d3f13dfe: Deleting consensus metadata
I20260620 06:14:46.006009 31602 ts_tablet_manager.cc:1918] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:14:46.006112 31602 log.cc:1196] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/eb785a60561b478393076e545a2ac8dd
I20260620 06:14:46.006210 31093 catalog_manager.cc:4841] TS e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): tablet eb785a60561b478393076e545a2ac8dd (table test-table [id=e8fdf95de66946c299ad457b888e10e3]) successfully deleted
I20260620 06:14:46.006456 31602 ts_tablet_manager.cc:1939] T eb785a60561b478393076e545a2ac8dd P 47c91e157eda41dc9d2b093403f73e7b: Deleting consensus metadata
I20260620 06:14:46.007517 31095 catalog_manager.cc:4841] TS 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): tablet eb785a60561b478393076e545a2ac8dd (table test-table [id=e8fdf95de66946c299ad457b888e10e3]) successfully deleted
W20260620 06:14:46.014472 31095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:32815 (error 108)
I20260620 06:14:46.015322 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31302
I20260620 06:14:46.015544 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:14:46.015159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:32815 (local address 127.0.0.1:57754)
0620 06:14:46.015305 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:14:46.015388 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 06:14:46.016187 31095 catalog_manager.cc:4584] TS a952cd6232ce4dbf9c2201461d4fee1d (127.14.209.129:32815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb785a60561b478393076e545a2ac8dd: Network error: Client connection negotiation failed: client connection to 127.14.209.129:32815: connect: Connection refused (error 111)
I20260620 06:14:46.021260 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31438
I20260620 06:14:46.027395 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31076
I20260620 06:15:06.033500 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40201
--webserver_interface=127.14.209.190
--webserver_port=42817
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:06.143276 31604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:06.143522 31604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:06.143594 31604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:06.147533 31604 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:15:06.147600 31604 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:15:06.147629 31604 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:15:06.147679 31604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:06.147722 31604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:06.147774 31604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:06.147810 31604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:06.147845 31604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:06.152879 31604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=42817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:06.153808 31604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:06.154755 31604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:06.162674 31612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.162824 31604 server_base.cc:1034] running on GCE node
W20260620 06:15:06.162752 31609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:06.162982 31610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.163552 31604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:06.164582 31604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:06.165791 31604 hybrid_clock.cc:648] HybridClock initialized: now 1781936106165782 us; error 27 us; skew 500 ppm
Jun 20 06:15:06 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936106, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:06.172359 31604 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:06.173251 31604 webserver.cc:460] Webserver started at http://127.14.209.190:42817/ using document root <none> and password file <none>
I20260620 06:15:06.173786 31604 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:06.173851 31604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:06.177774 31604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:15:06.181317 31619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.182523 31604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:06.182665 31604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "9f9ed32304c34cd29faa1e0bf957c59b"
format_stamp: "Formatted at 2026-06-20 06:14:23 on dist-test-slave-zt01"
server_key: "ef7020f978df0a194f87ddd92b27fdca"
server_key_iv: "c15398e4c7d4b78b5bbb07346f8744d0"
server_key_version: "encryptionkey@0"
I20260620 06:15:06.183213 31604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:06.195924 31604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:06.198839 31604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:06.198975 31604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:06.206382 31604 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40201
I20260620 06:15:06.206658 31671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40201 every 8 connection(s)
I20260620 06:15:06.207448 31604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:06.209473 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31604
I20260620 06:15:06.209919 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:32815
--local_ip_for_outbound_sockets=127.14.209.129
--tserver_master_addrs=127.14.209.190:40201
--webserver_port=46731
--webserver_interface=127.14.209.129
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:15:06.212018 31672 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:15:06.213550 31672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap starting.
I20260620 06:15:06.230276 31672 log.cc:824] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:06.241258 31672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:15:06.241611 31672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Bootstrap complete.
I20260620 06:15:06.245645 31672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:15:06.246021 31672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9ed32304c34cd29faa1e0bf957c59b, State: Initialized, Role: FOLLOWER
I20260620 06:15:06.246496 31672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:15:06.246603 31672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:06.246687 31672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:06.246819 31672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:15:06.248589 31672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:15:06.248914 31672 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9ed32304c34cd29faa1e0bf957c59b; no voters:
I20260620 06:15:06.249230 31672 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 06:15:06.249316 31676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:15:06.249523 31676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [term 3 LEADER]: Becoming Leader. State: Replica: 9f9ed32304c34cd29faa1e0bf957c59b, State: Running, Role: LEADER
I20260620 06:15:06.249801 31676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } }
I20260620 06:15:06.250398 31672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:06.250691 31676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9ed32304c34cd29faa1e0bf957c59b. Latest consensus state: current_term: 3 leader_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } } }
I20260620 06:15:06.250805 31676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:06.251170 31680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:06.251199 31677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ed32304c34cd29faa1e0bf957c59b" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40201 } } }
I20260620 06:15:06.251457 31677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:06.256035 31680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:06.256680 31680 catalog_manager.cc:1259] Loaded cluster ID: 5ca776d77abc47038f0f0aa50b088836
I20260620 06:15:06.256740 31680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:06.260030 31680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:06.261019 31680 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Loaded TSK: 0
I20260620 06:15:06.261101 31680 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f9ed32304c34cd29faa1e0bf957c59b: Loaded TSK: 1
F20260620 06:15:06.261838 31680 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe3ae8a79ad google::LogMessage::Fail() at ??:0
@ 0x7fe3ae8aa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe3ae8a74de google::LogMessage::Flush() at ??:0
@ 0x7fe3ae8a8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe3b156b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe3b1563f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe3b920c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe3b91e699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe3b91e7178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe3b9228df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe3b924782b std::function<>::operator()() at ??:0
@ 0x7fe3b91e6b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe3b91e7f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe3b91e4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe3b9228640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe3b0af93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe3b0af9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe3b0afb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe3b0aea512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe3b212e6db start_thread at ??:0
@ 0x7fe3ad3a561f clone at ??:0
W20260620 06:15:06.346865 31674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:06.347242 31674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:06.347311 31674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:06.353856 31674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:06.353912 31674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:06.353996 31674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:06.362219 31674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:32815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=46731
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:06.363487 31674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:06.364679 31674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:06.376349 31717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.376652 31674 server_base.cc:1034] running on GCE node
W20260620 06:15:06.377185 31720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:06.377671 31718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.378173 31674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:06.378917 31674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:06.380074 31674 hybrid_clock.cc:648] HybridClock initialized: now 1781936106380056 us; error 30 us; skew 500 ppm
Jun 20 06:15:06 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936106, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:06.383486 31674 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:15:06.384318 31674 webserver.cc:460] Webserver started at http://127.14.209.129:46731/ using document root <none> and password file <none>
I20260620 06:15:06.384855 31674 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:06.384956 31674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:06.388702 31674 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 06:15:06.390694 31729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.391942 31674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:06.392061 31674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "a952cd6232ce4dbf9c2201461d4fee1d"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "84fdb27cdab3b4ca5dad1eff0a4a7455"
server_key_iv: "532a43d63d68fbfb89a49fc8dcbd88a6"
server_key_version: "encryptionkey@0"
I20260620 06:15:06.392577 31674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:06.404500 31674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:06.407406 31674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:06.407582 31674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:06.408097 31674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:06.409070 31736 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:15:06.413975 31674 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:15:06.414045 31674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 06:15:06.414136 31674 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:15:06.415716 31674 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:15:06.415783 31674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:06.416052 31736 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap starting.
I20260620 06:15:06.425949 31674 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:32815
I20260620 06:15:06.426162 31843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:32815 every 8 connection(s)
I20260620 06:15:06.426892 31674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:06.436501 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31674
I20260620 06:15:06.436977 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:42509
--local_ip_for_outbound_sockets=127.14.209.130
--tserver_master_addrs=127.14.209.190:40201
--webserver_port=41489
--webserver_interface=127.14.209.130
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:15:06.437132 31736 log.cc:824] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:06.445602 31736 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:15:06.446012 31736 tablet_bootstrap.cc:492] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Bootstrap complete.
I20260620 06:15:06.447459 31736 ts_tablet_manager.cc:1397] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260620 06:15:06.450635 31736 raft_consensus.cc:348] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:06.451201 31736 raft_consensus.cc:729] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a952cd6232ce4dbf9c2201461d4fee1d, State: Initialized, Role: FOLLOWER
I20260620 06:15:06.451738 31736 consensus_queue.cc:260] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:06.452701 31736 ts_tablet_manager.cc:1428] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 06:15:06.510605 31846 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:06.428781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:45021)
0620 06:15:06.429236 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.429285 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:06.430276 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:06.509663 (+ 79387us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
W20260620 06:15:06.511149 31844 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:40201 (0 consecutive failures): Network error: Failed to ping master at 127.14.209.190:40201: Client connection negotiation failed: client connection to 127.14.209.190:40201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:15:06.511929 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.511624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:51391)
0620 06:15:06.511776 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.511855 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:15:06.512629 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.512368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:33125)
0620 06:15:06.512497 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.512560 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:15:06.512843 31844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:15:06.545786 31848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:06.545989 31848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:06.546062 31848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:06.549991 31848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:06.550041 31848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:06.550143 31848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:06.555241 31848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:42509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=41489
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:06.556113 31848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:06.556977 31848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:06.565773 31854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:06.565820 31855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.565953 31848 server_base.cc:1034] running on GCE node
W20260620 06:15:06.566057 31857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.566371 31848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:06.566984 31848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:06.568173 31848 hybrid_clock.cc:648] HybridClock initialized: now 1781936106568152 us; error 31 us; skew 500 ppm
Jun 20 06:15:06 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936106, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:06.570689 31848 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:15:06.571575 31848 webserver.cc:460] Webserver started at http://127.14.209.130:41489/ using document root <none> and password file <none>
I20260620 06:15:06.572068 31848 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:06.572155 31848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:06.575664 31848 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:06.577548 31864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.578766 31848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:06.578874 31848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "e602c04a88cd4c72881e6b79d3f13dfe"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "aefae16997661653fa5ddfe0e8eda351"
server_key_iv: "d7519d87d17c9bac1d0c0d976431eb29"
server_key_version: "encryptionkey@0"
I20260620 06:15:06.579408 31848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:06.607035 31848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:06.609758 31848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:06.609954 31848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:06.610487 31848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:06.611303 31848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:06.611349 31848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.611387 31848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:06.611440 31848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.620826 31848 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:42509
I20260620 06:15:06.620857 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:42509 every 8 connection(s)
I20260620 06:15:06.621728 31848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:06.624130 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31848
I20260620 06:15:06.624583 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:45149
--local_ip_for_outbound_sockets=127.14.209.131
--tserver_master_addrs=127.14.209.190:40201
--webserver_port=41179
--webserver_interface=127.14.209.131
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:15:06.625599 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.623798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:59403)
0620 06:15:06.624385 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.624951 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":146,"threads_started":1}
W20260620 06:15:06.626226 31978 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:40201 (0 consecutive failures): Network error: Failed to ping master at 127.14.209.190:40201: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
I20260620 06:15:06.626806 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.626585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:33515)
0620 06:15:06.626701 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.626755 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 06:15:06.627568 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.627300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:56427)
0620 06:15:06.627430 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.627489 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:15:06.627748 31978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:15:06.729487 31981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:06.729704 31981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:06.729744 31981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:06.733893 31981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:06.733942 31981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:06.734030 31981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:06.739176 31981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:45149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=41179
--tserver_master_addrs=127.14.209.190:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:06.740015 31981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:06.740890 31981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:06.749922 31987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:06.749902 31989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:06.749890 31986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:06.750166 31981 server_base.cc:1034] running on GCE node
I20260620 06:15:06.750510 31981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:06.751176 31981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:06.752350 31981 hybrid_clock.cc:648] HybridClock initialized: now 1781936106752344 us; error 26 us; skew 500 ppm
Jun 20 06:15:06 dist-test-slave-zt01 krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936106, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:06.754969 31981 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:15:06.755751 31981 webserver.cc:460] Webserver started at http://127.14.209.131:41179/ using document root <none> and password file <none>
I20260620 06:15:06.756258 31981 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:06.756345 31981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:06.759802 31981 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:06.761798 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.763041 31981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:06.763203 31981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "47c91e157eda41dc9d2b093403f73e7b"
format_stamp: "Formatted at 2026-06-20 06:14:24 on dist-test-slave-zt01"
server_key: "36e2b8d98e765fbbd0370fa043783d27"
server_key_iv: "edbd631bec6104e3120a93caf295ef03"
server_key_version: "encryptionkey@0"
I20260620 06:15:06.763680 31981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:06.777222 31981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:06.779922 31981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:06.780103 31981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:06.780690 31981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:06.781421 31981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:06.781471 31981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.781538 31981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:06.781574 31981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:06.790576 31981 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:45149
I20260620 06:15:06.790630 32109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:45149 every 8 connection(s)
I20260620 06:15:06.791494 31981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:06.794891 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.793363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:48207)
0620 06:15:06.793772 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.793867 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
W20260620 06:15:06.795383 32110 heartbeater.cc:644] Failed to heartbeat to 127.14.209.190:40201 (0 consecutive failures): Network error: Failed to ping master at 127.14.209.190:40201: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
I20260620 06:15:06.795954 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.795770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:44547)
0620 06:15:06.795864 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.795906 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 06:15:06.796618 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:06.796362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:35523)
0620 06:15:06.796505 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:06.796545 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 06:15:06.796794 32110 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:15:06.800897 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 31981
I20260620 06:15:07.513820 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.513386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:41201)
0620 06:15:07.513652 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:07.513712 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260620 06:15:07.628635 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.628260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:54347)
0620 06:15:07.628481 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:07.628540 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260620 06:15:07.797618 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.797264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:58681)
0620 06:15:07.797481 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:07.797531 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260620 06:15:07.884783 32117 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:07.884999 32117 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:07.886130 32117 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
Jun 20 06:15:07 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936106, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.130@KRBTEST.COM
Jun 20 06:15:07 dist-test-slave-zt01 krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936106, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.131@KRBTEST.COM
I20260620 06:15:07.900481 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.886314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:42509 (local address 127.14.209.129:38931)
0620 06:15:07.886482 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:07.886496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:07.886615 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:07.888651 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:07.888655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:07.889009 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:07.889460 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:07.889474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.891378 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.891380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:07.892227 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:07.892236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.892965 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.892969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:07.893174 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:07.894076 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:07.894088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:07.895859 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:07.898204 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:07.898210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:07.898223 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:07.898464 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:07.898810 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:07.898815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:07.898817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:07.898947 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:07.899563 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:07.899569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:07.899787 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 06:15:07.899972 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 06:15:07.900292 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":166,"spinlock_wait_cycles":768}
I20260620 06:15:07.900494 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.886527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:38931 (local address 127.14.209.130:42509)
0620 06:15:07.886864 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:07.886891 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:07.886932 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:15:07.887847 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:07.887854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:07.888278 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:07.888595 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:07.889639 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.891200 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.892397 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.892838 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.893022 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:07.896168 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:07.896177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:07.896188 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:07.896368 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:07.898079 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:07.898587 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:07.898593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:07.898597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:07.898711 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:07.899072 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:07.899076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:07.899078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:07.899436 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:07.899589 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:07.900079 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 06:15:07.900322 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":60,"threads_started":1}
I20260620 06:15:07.900534 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.886500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:36443 (local address 127.14.209.131:45149)
0620 06:15:07.886875 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:07.886893 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:07.886932 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:07.887842 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:07.887850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:07.888279 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:07.888595 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:07.889642 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.891105 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.892374 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.892840 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.893017 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:07.896442 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:07.896446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:07.896456 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:07.896481 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:07.898308 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:07.898752 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:07.898757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:07.898764 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:07.898868 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:07.899172 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:07.899176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:07.899178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:07.899518 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:07.899655 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:07.900076 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 06:15:07.900347 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":88,"threads_started":1}
I20260620 06:15:07.900537 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:07.886337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45149 (local address 127.14.209.129:36443)
0620 06:15:07.886657 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:07.886671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:07.886772 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:07.889058 (+ 2286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:07.889060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:07.889213 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:07.889477 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:07.889484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.891262 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.891268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:07.892226 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:07.892235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:07.892968 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:07.892971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:07.893177 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:07.894018 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:07.894034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:07.896256 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:07.898442 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:07.898445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:07.898447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:07.898642 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:07.898965 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:07.898970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:07.898972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:07.899096 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:07.899635 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:07.899638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:07.899791 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 06:15:07.899972 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 06:15:07.900376 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":255,"spinlock_wait_cycles":3328,"thread_start_us":58,"threads_started":1}
W20260620 06:15:07.903398 31730 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
W20260620 06:15:07.903564 31732 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
I20260620 06:15:07.903666 31732 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d; no voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe
I20260620 06:15:07.903865 32117 raft_consensus.cc:2726] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:15:08.514966 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:08.514569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:39773)
0620 06:15:08.514814 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:08.514865 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260620 06:15:08.629663 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:08.629289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:41425)
0620 06:15:08.629521 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:08.629580 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260620 06:15:08.798682 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:08.798291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:55641)
0620 06:15:08.798513 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:08.798606 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260620 06:15:09.516242 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:09.515839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:58631)
0620 06:15:09.516082 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:09.516141 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260620 06:15:09.630682 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:09.630317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:37373)
0620 06:15:09.630535 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:09.630593 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260620 06:15:09.799674 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:09.799332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:46213)
0620 06:15:09.799535 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:09.799583 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20260620 06:15:10.039172 32127 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:10.039332 32127 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:10.039770 32127 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
W20260620 06:15:10.040494 31730 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
W20260620 06:15:10.040670 31732 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
I20260620 06:15:10.040741 31732 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d; no voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe
I20260620 06:15:10.040933 32127 raft_consensus.cc:2726] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:15:10.517414 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:10.516997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:33343)
0620 06:15:10.517263 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:10.517320 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260620 06:15:10.631855 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:10.631502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:57503)
0620 06:15:10.631727 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:10.631777 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260620 06:15:10.800750 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:10.800342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:49309)
0620 06:15:10.800611 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:10.800664 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260620 06:15:11.518745 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:11.518341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:51445)
0620 06:15:11.518605 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:11.518655 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":69,"threads_started":1}
I20260620 06:15:11.632876 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:11.632516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:60983)
0620 06:15:11.632740 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:11.632793 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260620 06:15:11.801960 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:11.801474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:41625)
0620 06:15:11.801807 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:11.801873 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260620 06:15:11.837165 32134 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:11.837321 32134 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:11.837785 32134 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
W20260620 06:15:11.838505 31732 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
W20260620 06:15:11.838578 31730 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
I20260620 06:15:11.838621 31732 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d; no voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe
I20260620 06:15:11.838765 32134 raft_consensus.cc:2726] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:15:12.520011 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:12.519561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:46055)
0620 06:15:12.519869 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:12.519921 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260620 06:15:12.633893 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:12.633532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:34247)
0620 06:15:12.633764 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:12.633815 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260620 06:15:12.803014 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:12.802661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:39543)
0620 06:15:12.802882 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:12.802932 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260620 06:15:13.521239 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:13.520791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:55433)
0620 06:15:13.521093 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:13.521146 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
I20260620 06:15:13.634990 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:13.634582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:35883)
0620 06:15:13.634847 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:13.634898 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260620 06:15:13.804306 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:13.803890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:56381)
0620 06:15:13.804160 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:13.804212 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":65,"threads_started":1}
I20260620 06:15:14.522457 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:14.522063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:43495)
0620 06:15:14.522314 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:14.522366 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260620 06:15:14.636106 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:14.635705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:34395)
0620 06:15:14.635951 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:14.636005 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260620 06:15:14.805380 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:14.805003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:38869)
0620 06:15:14.805248 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:14.805301 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":65,"threads_started":1}
I20260620 06:15:14.942901 32144 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:14.943024 32144 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:14.943534 32144 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
W20260620 06:15:14.944248 31730 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
W20260620 06:15:14.944382 31732 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
I20260620 06:15:14.944473 31732 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d; no voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe
I20260620 06:15:14.944636 32144 raft_consensus.cc:2726] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:15:15.523627 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:15.523230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:45979)
0620 06:15:15.523471 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:15.523525 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260620 06:15:15.637128 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:15.636776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:54425)
0620 06:15:15.636999 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:15.637050 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":67,"threads_started":1}
I20260620 06:15:15.806540 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:15.806103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:46481)
0620 06:15:15.806389 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:15.806450 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20260620 06:15:16.524878 32148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:16.524455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:52975)
0620 06:15:16.524729 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:16.524784 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260620 06:15:16.638314 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:16.637918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:33231)
0620 06:15:16.638181 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:16.638231 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
I20260620 06:15:16.807695 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:16.807305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:38791)
0620 06:15:16.807551 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:16.807604 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260620 06:15:16.834543 32151 raft_consensus.cc:482] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:16.834743 32151 raft_consensus.cc:504] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e602c04a88cd4c72881e6b79d3f13dfe" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 42509 } } peers { permanent_uuid: "47c91e157eda41dc9d2b093403f73e7b" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45149 } } peers { permanent_uuid: "a952cd6232ce4dbf9c2201461d4fee1d" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 32815 } }
I20260620 06:15:16.835227 32151 leader_election.cc:288] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509), 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149)
W20260620 06:15:16.835960 31732 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47c91e157eda41dc9d2b093403f73e7b (127.14.209.131:45149): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
W20260620 06:15:16.836053 31730 leader_election.cc:341] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e602c04a88cd4c72881e6b79d3f13dfe (127.14.209.130:42509): Not found: Tablet not found: eb785a60561b478393076e545a2ac8dd
I20260620 06:15:16.836166 31730 leader_election.cc:302] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a952cd6232ce4dbf9c2201461d4fee1d; no voters: 47c91e157eda41dc9d2b093403f73e7b, e602c04a88cd4c72881e6b79d3f13dfe
I20260620 06:15:16.836330 32151 raft_consensus.cc:2726] T eb785a60561b478393076e545a2ac8dd P a952cd6232ce4dbf9c2201461d4fee1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:15:17.526019 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:17.525610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:46825)
0620 06:15:17.525869 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:17.525925 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260620 06:15:17.639389 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:17.638988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:44685)
0620 06:15:17.639254 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:17.639308 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
I20260620 06:15:17.808696 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:17.808345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:54781)
0620 06:15:17.808564 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:17.808615 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260620 06:15:18.527185 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:18.526718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:34409)
0620 06:15:18.526986 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:18.527039 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
I20260620 06:15:18.640406 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:18.640038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:57615)
0620 06:15:18.640270 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:18.640325 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260620 06:15:18.809725 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:18.809352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:46949)
0620 06:15:18.809591 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:18.809643 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260620 06:15:19.528393 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:19.527974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:50827)
0620 06:15:19.528241 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:19.528297 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260620 06:15:19.641451 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:19.641090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:50979)
0620 06:15:19.641320 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:19.641372 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260620 06:15:19.810868 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:19.810422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:37301)
0620 06:15:19.810722 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:19.810777 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":70,"threads_started":1}
I20260620 06:15:20.529495 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:20.529112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:36347)
0620 06:15:20.529346 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:20.529399 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260620 06:15:20.642505 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:20.642110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:49541)
0620 06:15:20.642372 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:20.642423 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260620 06:15:20.812047 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:20.811691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:34595)
0620 06:15:20.811914 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:20.811964 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260620 06:15:21.530735 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:21.530370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.129:43235)
0620 06:15:21.530597 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:21.530646 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260620 06:15:21.643725 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:21.643387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.130:49961)
0620 06:15:21.643594 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:21.643643 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260620 06:15:21.810421 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31674
I20260620 06:15:21.813210 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:21.812847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40201 (local address 127.14.209.131:42251)
0620 06:15:21.813069 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:21.813119 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.190:40201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260620 06:15:21.816326 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31848
I20260620 06:15:21.821570 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 31981
2026-06-20T06:15:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59207 ms)
[----------] 30 tests from SecurityITest (236102 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:21 dist-test-slave-zt01 krb5kdc[32169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:21 dist-test-slave-zt01 krb5kdc[32169](info): set up 2 sockets
Jun 20 06:15:21 dist-test-slave-zt01 krb5kdc[32169](info): commencing operation
krb5kdc: starting...
W20260620 06:15:22.862975 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:22 dist-test-slave-zt01 krb5kdc[32169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:22.979701 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40171
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:23.083767 32182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:23.083998 32182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:23.084038 32182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:23.087986 32182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:23.088034 32182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:23.088053 32182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:23.088070 32182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:23.088085 32182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:23.093086 32182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:23.093997 32182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:23.094915 32182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:23.103569 32187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.103711 32182 server_base.cc:1034] running on GCE node
W20260620 06:15:23.103773 32188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:23.103873 32190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.104400 32182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:23.105396 32182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:23.106578 32182 hybrid_clock.cc:648] HybridClock initialized: now 1781936123106552 us; error 45 us; skew 500 ppm
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:23.113220 32182 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:23.114066 32182 webserver.cc:460] Webserver started at http://127.14.209.190:43479/ using document root <none> and password file <none>
I20260620 06:15:23.114598 32182 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:23.114655 32182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:23.114862 32182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:23.116858 32182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "46fc0a47ef944eea9405d04a2c5818ac"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "ce8c937ebf4af38e3b27f68e972b99ab"
server_key_iv: "c6a3ac67beb954e08ef19f3b2ef64f7c"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.117398 32182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "46fc0a47ef944eea9405d04a2c5818ac"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "ce8c937ebf4af38e3b27f68e972b99ab"
server_key_iv: "c6a3ac67beb954e08ef19f3b2ef64f7c"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.120839 32182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:23.123036 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.124680 32182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:23.124814 32182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "46fc0a47ef944eea9405d04a2c5818ac"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "ce8c937ebf4af38e3b27f68e972b99ab"
server_key_iv: "c6a3ac67beb954e08ef19f3b2ef64f7c"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.124922 32182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:23.140322 32182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:23.142925 32182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:23.143143 32182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:23.149765 32182 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40171
I20260620 06:15:23.149796 32249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40171 every 8 connection(s)
I20260620 06:15:23.150604 32182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:23.153223 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:23.154951 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 32182
I20260620 06:15:23.155112 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:23.155406 15174 external_mini_cluster.cc:1428] Setting key e4a6b9549560d9a4110ddca4bd01b381
I20260620 06:15:23.159034 32250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac: Bootstrap starting.
I20260620 06:15:23.161422 32250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:23.162304 32250 log.cc:824] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:23.164242 32250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac: No bootstrap required, opened a new log
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:23.168455 32250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fc0a47ef944eea9405d04a2c5818ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40171 } }
I20260620 06:15:23.168723 32250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:23.168782 32250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fc0a47ef944eea9405d04a2c5818ac, State: Initialized, Role: FOLLOWER
I20260620 06:15:23.169217 32250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fc0a47ef944eea9405d04a2c5818ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40171 } }
I20260620 06:15:23.169381 32250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:23.169477 32250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:23.169587 32250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:23.170614 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.156728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.14.209.190:40171)
0620 06:15:23.157144 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.157290 (+ 146us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.157425 (+ 135us) server_negotiation.cc:407] Connection header received
0620 06:15:23.158262 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.158279 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.158573 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.158950 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:23.159764 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.160743 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.161620 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.162309 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.162475 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.165095 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:23.165101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:23.165115 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:23.165143 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:23.167695 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.168125 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.168131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.168136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.168273 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.168534 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.168538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.168539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.168993 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:23.169187 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.169415 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.169703 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260620 06:15:23.170583 32250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fc0a47ef944eea9405d04a2c5818ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40171 } }
I20260620 06:15:23.170949 32250 leader_election.cc:302] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46fc0a47ef944eea9405d04a2c5818ac; no voters:
I20260620 06:15:23.171268 32250 leader_election.cc:288] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:23.171397 32255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:23.171650 32255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [term 1 LEADER]: Becoming Leader. State: Replica: 46fc0a47ef944eea9405d04a2c5818ac, State: Running, Role: LEADER
I20260620 06:15:23.172032 32255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fc0a47ef944eea9405d04a2c5818ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40171 } }
I20260620 06:15:23.172422 32250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:23.173614 32256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46fc0a47ef944eea9405d04a2c5818ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fc0a47ef944eea9405d04a2c5818ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40171 } } }
I20260620 06:15:23.173743 32256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:23.173998 32256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46fc0a47ef944eea9405d04a2c5818ac. Latest consensus state: current_term: 1 leader_uuid: "46fc0a47ef944eea9405d04a2c5818ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fc0a47ef944eea9405d04a2c5818ac" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40171 } } }
I20260620 06:15:23.174074 32256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:23.174137 32264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:23.177997 32264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:23.182935 32264 catalog_manager.cc:1347] Generated new cluster ID: ee48200ce1654477a3046f14f7094f5d
I20260620 06:15:23.182997 32264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:23.203804 32264 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:23.204670 32264 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:23.209064 32264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac: Generated new TSK 0
I20260620 06:15:23.209669 32264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:15:23.241674 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40171
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:23.361441 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:23.361658 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:23.361706 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:23.365463 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:23.365505 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:23.365561 32276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:23.370505 32276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:23.371340 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:23.372174 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:23.381434 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.381567 32276 server_base.cc:1034] running on GCE node
W20260620 06:15:23.381405 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:23.381506 32281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.382078 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:23.382679 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:23.383867 32276 hybrid_clock.cc:648] HybridClock initialized: now 1781936123383848 us; error 31 us; skew 500 ppm
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:23.386539 32276 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:15:23.387339 32276 webserver.cc:460] Webserver started at http://127.14.209.129:34067/ using document root <none> and password file <none>
I20260620 06:15:23.387818 32276 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:23.387872 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:23.388018 32276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:23.389848 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "11ab56a2129541a5881fbbe402940de4"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "ad5eab507903141d1be3c6488be709dd"
server_key_iv: "4d5f2b57637c74937390a3f82ebff63e"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.390317 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "11ab56a2129541a5881fbbe402940de4"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "ad5eab507903141d1be3c6488be709dd"
server_key_iv: "4d5f2b57637c74937390a3f82ebff63e"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.393766 32276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:15:23.395977 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.397081 32276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:23.397179 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "11ab56a2129541a5881fbbe402940de4"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "ad5eab507903141d1be3c6488be709dd"
server_key_iv: "4d5f2b57637c74937390a3f82ebff63e"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.397296 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:23.414111 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:23.416810 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:23.416986 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:23.417505 32276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:23.418241 32276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:23.418285 32276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.418349 32276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:23.418380 32276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.426582 32276 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:45581
I20260620 06:15:23.426620 32404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:45581 every 8 connection(s)
I20260620 06:15:23.427385 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:23.427820 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 32276
I20260620 06:15:23.427953 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:15:23.428237 15174 external_mini_cluster.cc:1428] Setting key 8774817a53293e3731c9ec62a1cd23f7
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:15:23.442448 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.429068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:59535 (local address 127.14.209.190:40171)
0620 06:15:23.429206 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.429210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.430135 (+ 925us) server_negotiation.cc:407] Connection header received
0620 06:15:23.431661 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.431664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.431707 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.431777 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:23.433242 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.433855 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.435688 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.435915 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.435996 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.438633 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:23.438636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:23.438638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:23.438660 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:23.440169 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.440758 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.440761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.440762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.440840 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.441269 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.441272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.441274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.441474 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:23.441566 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.442224 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.442330 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:15:23.443620 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.429519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40171 (local address 127.14.209.129:59535)
0620 06:15:23.429981 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:23.430029 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:23.431441 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:23.431943 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:23.431954 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:23.432455 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:23.433089 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.433110 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.433986 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.433990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.435560 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.435584 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.436065 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.436071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.436301 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.436882 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:23.436901 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:23.438491 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:23.440328 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:23.440335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:23.440353 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:23.440643 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:23.440963 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:23.440969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:23.440974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:23.441163 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:23.441587 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:23.441596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:23.441938 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 06:15:23.442188 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 06:15:23.442547 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
I20260620 06:15:23.445030 32405 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40171
I20260620 06:15:23.445253 32405 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:23.445696 32405 heartbeater.cc:511] Master 127.14.209.190:40171 requested a full tablet report, sending...
I20260620 06:15:23.447342 32214 ts_manager.cc:194] Registered new tserver with Master: 11ab56a2129541a5881fbbe402940de4 (127.14.209.129:45581)
I20260620 06:15:23.448983 32214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:59535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:15:23.456513 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40171
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:23.561501 32410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:23.561714 32410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:23.561755 32410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:23.565608 32410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:23.565654 32410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:23.565748 32410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:23.570788 32410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:23.571678 32410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:23.572646 32410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:23.581575 32416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.581647 32410 server_base.cc:1034] running on GCE node
W20260620 06:15:23.581597 32415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:23.581595 32418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.582203 32410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:23.582758 32410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:23.583920 32410 hybrid_clock.cc:648] HybridClock initialized: now 1781936123583908 us; error 23 us; skew 500 ppm
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:23.586374 32410 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:15:23.587131 32410 webserver.cc:460] Webserver started at http://127.14.209.130:45661/ using document root <none> and password file <none>
I20260620 06:15:23.587625 32410 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:23.587685 32410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:23.587855 32410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:23.589597 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "45f0c433901449ada80a34d007ea279a"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "a454c393ce25d88d88c01df51b42700c"
server_key_iv: "7f3c782af658991dd8d2d6da53382624"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.590065 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "45f0c433901449ada80a34d007ea279a"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "a454c393ce25d88d88c01df51b42700c"
server_key_iv: "7f3c782af658991dd8d2d6da53382624"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.593397 32410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:23.595467 32425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.596567 32410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:23.596655 32410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "45f0c433901449ada80a34d007ea279a"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "a454c393ce25d88d88c01df51b42700c"
server_key_iv: "7f3c782af658991dd8d2d6da53382624"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.596771 32410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:23.615864 32410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:23.618777 32410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:23.618976 32410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:23.619560 32410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:23.620344 32410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:23.620389 32410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.620453 32410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:23.620491 32410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.629479 32410 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:46517
I20260620 06:15:23.629513 32538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:46517 every 8 connection(s)
I20260620 06:15:23.630303 32410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:23.632114 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 32410
I20260620 06:15:23.632252 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:15:23.632573 15174 external_mini_cluster.cc:1428] Setting key 8e7ee9b9e40ff2a7a2ea37df31685a26
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:15:23.644456 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.632138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:53485 (local address 127.14.209.190:40171)
0620 06:15:23.632256 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.632260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.633531 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 06:15:23.634957 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.634963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.635024 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.635153 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:23.636624 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.637143 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.638261 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.638488 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.638591 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.640989 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:23.640993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:23.640995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:23.641024 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:23.642592 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.643050 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.643094 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.643096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.643174 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.643490 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.643493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.643494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.643695 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:23.643772 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.644226 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.644339 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
I20260620 06:15:23.645357 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.632623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40171 (local address 127.14.209.130:53485)
0620 06:15:23.633318 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:23.633409 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:23.634763 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:23.635279 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:23.635285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:23.635819 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:23.636471 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.636493 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.637245 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.637248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.638121 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.638131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.638634 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.638638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.638796 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.639388 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:23.639403 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:23.640833 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:23.642718 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:23.642722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:23.642732 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:23.642964 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:23.643243 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:23.643245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:23.643247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:23.643368 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:23.643788 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:23.643793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:23.643999 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 06:15:23.644157 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 06:15:23.644383 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":138,"threads_started":1}
I20260620 06:15:23.646473 32539 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40171
I20260620 06:15:23.646685 32539 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:23.647195 32539 heartbeater.cc:511] Master 127.14.209.190:40171 requested a full tablet report, sending...
I20260620 06:15:23.648202 32214 ts_manager.cc:194] Registered new tserver with Master: 45f0c433901449ada80a34d007ea279a (127.14.209.130:46517)
I20260620 06:15:23.649062 32214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:53485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:15:23.661842 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40171
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:23.767643 32544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:23.767920 32544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:23.767964 32544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:23.771888 32544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:23.771934 32544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:23.771987 32544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:23.777192 32544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:23.778110 32544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:23.779001 32544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:23.787791 32550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:23.787815 32549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:23.787827 32552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:23.787905 32544 server_base.cc:1034] running on GCE node
I20260620 06:15:23.788347 32544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:23.788928 32544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:23.790124 32544 hybrid_clock.cc:648] HybridClock initialized: now 1781936123790114 us; error 27 us; skew 500 ppm
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:23.792649 32544 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:15:23.793402 32544 webserver.cc:460] Webserver started at http://127.14.209.131:44541/ using document root <none> and password file <none>
I20260620 06:15:23.793900 32544 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:23.793978 32544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:23.794185 32544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:23.796046 32544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "ffdbddbf278140169abcbdc34f02def1"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "685a9059c42303b9fd7f2a42bdbd4287"
server_key_iv: "077b7f9d9410092aa859a065a6e254d0"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.796558 32544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "ffdbddbf278140169abcbdc34f02def1"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "685a9059c42303b9fd7f2a42bdbd4287"
server_key_iv: "077b7f9d9410092aa859a065a6e254d0"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.799981 32544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:23.802543 32559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.803941 32544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:15:23.804081 32544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "ffdbddbf278140169abcbdc34f02def1"
format_stamp: "Formatted at 2026-06-20 06:15:23 on dist-test-slave-zt01"
server_key: "685a9059c42303b9fd7f2a42bdbd4287"
server_key_iv: "077b7f9d9410092aa859a065a6e254d0"
server_key_version: "encryptionkey@0"
I20260620 06:15:23.804188 32544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:23.820282 32544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:23.823096 32544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:23.823282 32544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:23.823827 32544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:23.824595 32544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:23.824667 32544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.824736 32544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:23.824791 32544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:23.833871 32544 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:40319
I20260620 06:15:23.833902 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:40319 every 8 connection(s)
I20260620 06:15:23.834713 32544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:23.837553 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 32544
I20260620 06:15:23.837688 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:15:23.838009 15174 external_mini_cluster.cc:1428] Setting key 4270ba73ee092993d7550068979768ad
Jun 20 06:15:23 dist-test-slave-zt01 krb5kdc[32169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:23.849164 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.836454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:51075 (local address 127.14.209.190:40171)
0620 06:15:23.836603 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.836608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.837632 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 06:15:23.839829 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.839834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.839888 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.839978 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:23.841076 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.841913 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.843190 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.843438 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.843514 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.845823 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:23.845826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:23.845828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:23.845860 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:23.847193 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.847715 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.847718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.847719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.847798 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.848195 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.848198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.848200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.848390 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:23.848473 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.848956 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.849050 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:15:23.849857 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.836865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40171 (local address 127.14.209.131:51075)
0620 06:15:23.837445 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:23.837504 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:23.839153 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:23.840165 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:23.840172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:23.840508 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:23.840949 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.840960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.842039 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.842042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.842925 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.842934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.843570 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.843573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.843707 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.844285 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:23.844300 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:23.845647 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:23.847308 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:23.847313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:23.847322 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:23.847574 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:23.847936 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:23.847938 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:23.847940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:23.848076 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:23.848518 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:23.848523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:23.848738 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 06:15:23.848892 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 06:15:23.849110 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260620 06:15:23.850935 32673 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40171
I20260620 06:15:23.851189 32673 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:23.851655 32673 heartbeater.cc:511] Master 127.14.209.190:40171 requested a full tablet report, sending...
I20260620 06:15:23.852568 32214 ts_manager.cc:194] Registered new tserver with Master: ffdbddbf278140169abcbdc34f02def1 (127.14.209.131:40319)
I20260620 06:15:23.853094 32214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:51075
I20260620 06:15:23.862833 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:15:23.871381 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.864686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.14.209.190:40171)
0620 06:15:23.864808 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.864813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.864976 (+ 163us) server_negotiation.cc:407] Connection header received
0620 06:15:23.865066 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.865068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.865103 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.865170 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:23.865754 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.866217 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.866956 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.867199 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.867296 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.868550 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:23.868553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:23.868555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:23.868574 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:23.869965 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.870320 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.870322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.870323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.870390 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:23.870596 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:23.870598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:23.870599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:23.870801 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:23.870899 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.871116 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.871200 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:15:23.874001 32214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:15:23.876962 32214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:15:23.890708 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.884840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45581 (local address 127.0.0.1:59078)
0620 06:15:23.885608 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:23.885621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:23.885696 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:23.886710 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:23.886713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:23.886732 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:23.886839 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.886845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.887961 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.887966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.889218 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.889228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.890174 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.890177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.890283 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.890336 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 06:15:23.890458 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 06:15:23.890568 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":53,"thread_start_us":87,"threads_started":1}
I20260620 06:15:23.890717 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.885792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.14.209.129:45581)
0620 06:15:23.886090 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.886101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.886122 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:15:23.886196 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.886202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.886388 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.886544 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:23.886955 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.887837 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.889358 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.890055 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.890186 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.890306 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.890487 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.890581 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":135,"threads_started":1}
I20260620 06:15:23.890843 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.884890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.14.209.131:40319)
0620 06:15:23.885281 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.885295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.885707 (+ 412us) server_negotiation.cc:407] Connection header received
0620 06:15:23.885796 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.885801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.885953 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.886098 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:23.886407 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.887829 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.889326 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.890047 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.890136 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.890261 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.890606 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.890700 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":133,"threads_started":1}
I20260620 06:15:23.891260 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.885391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46517 (local address 127.0.0.1:53982)
0620 06:15:23.886340 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:23.886355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:23.886431 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:23.887101 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:23.887104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:23.887123 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:23.887229 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.887235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.888544 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.888547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.889657 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.889666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.890867 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.890871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.890979 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.891006 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:15:23.891110 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 06:15:23.891163 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":103,"threads_started":1}
I20260620 06:15:23.891261 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.884737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:40319 (local address 127.0.0.1:54170)
0620 06:15:23.885393 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:23.885438 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:23.885568 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:23.886099 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:23.886105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:23.886126 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:23.886267 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.886275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.887981 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.887984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.889177 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:23.889185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.890359 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.890363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:23.890451 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.890472 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:15:23.891007 (+ 535us) client_negotiation.cc:240] Negotiation successful
0620 06:15:23.891121 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":38,"thread_start_us":55,"threads_started":1}
I20260620 06:15:23.891417 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:23.885567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.14.209.130:46517)
0620 06:15:23.886662 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:23.886673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:23.886693 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:15:23.886771 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:23.886775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:23.886929 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:23.887084 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:23.887345 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.888404 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.889786 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:23.890586 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:23.890689 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:23.890800 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:23.891157 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 06:15:23.891255 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":128,"threads_started":1}
I20260620 06:15:23.893962 32339 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5779b6ee2b49938194ab03be999cda (DEFAULT_TABLE table=test-table [id=657d26ee44d24841ae06103c483aab23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:23.894394 32607 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5779b6ee2b49938194ab03be999cda (DEFAULT_TABLE table=test-table [id=657d26ee44d24841ae06103c483aab23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:23.895018 32339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5779b6ee2b49938194ab03be999cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:23.894677 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5779b6ee2b49938194ab03be999cda (DEFAULT_TABLE table=test-table [id=657d26ee44d24841ae06103c483aab23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:23.895385 32607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5779b6ee2b49938194ab03be999cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:23.895649 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5779b6ee2b49938194ab03be999cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:23.901988 32691 tablet_bootstrap.cc:492] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Bootstrap starting.
I20260620 06:15:23.902910 32690 tablet_bootstrap.cc:492] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: Bootstrap starting.
I20260620 06:15:23.903316 32692 tablet_bootstrap.cc:492] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Bootstrap starting.
I20260620 06:15:23.904464 32691 tablet_bootstrap.cc:654] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:23.904917 32692 tablet_bootstrap.cc:654] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:23.904909 32690 tablet_bootstrap.cc:654] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:23.905781 32691 log.cc:824] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:23.906031 32690 log.cc:824] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:23.906188 32692 log.cc:824] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:23.907869 32691 tablet_bootstrap.cc:492] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: No bootstrap required, opened a new log
I20260620 06:15:23.907799 32690 tablet_bootstrap.cc:492] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: No bootstrap required, opened a new log
I20260620 06:15:23.908082 32691 ts_tablet_manager.cc:1397] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 06:15:23.908082 32690 ts_tablet_manager.cc:1397] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:15:23.908334 32692 tablet_bootstrap.cc:492] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: No bootstrap required, opened a new log
I20260620 06:15:23.908512 32692 ts_tablet_manager.cc:1397] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:15:23.911201 32692 raft_consensus.cc:348] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:23.911432 32692 raft_consensus.cc:374] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:23.911505 32692 raft_consensus.cc:729] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffdbddbf278140169abcbdc34f02def1, State: Initialized, Role: FOLLOWER
I20260620 06:15:23.911924 32692 consensus_queue.cc:260] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:23.911916 32690 raft_consensus.cc:348] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:23.911916 32691 raft_consensus.cc:348] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:23.912165 32691 raft_consensus.cc:374] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:23.912242 32690 raft_consensus.cc:374] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:23.912297 32690 raft_consensus.cc:729] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ab56a2129541a5881fbbe402940de4, State: Initialized, Role: FOLLOWER
I20260620 06:15:23.912297 32691 raft_consensus.cc:729] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0c433901449ada80a34d007ea279a, State: Initialized, Role: FOLLOWER
I20260620 06:15:23.912706 32690 consensus_queue.cc:260] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:23.912706 32691 consensus_queue.cc:260] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:23.912817 32692 ts_tablet_manager.cc:1428] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 06:15:23.913193 32673 heartbeater.cc:503] Master 127.14.209.190:40171 was elected leader, sending a full tablet report...
I20260620 06:15:23.913292 32539 heartbeater.cc:503] Master 127.14.209.190:40171 was elected leader, sending a full tablet report...
I20260620 06:15:23.913605 32691 ts_tablet_manager.cc:1428] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:15:23.913605 32690 ts_tablet_manager.cc:1428] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:15:23.914155 32405 heartbeater.cc:503] Master 127.14.209.190:40171 was elected leader, sending a full tablet report...
W20260620 06:15:23.928478 32406 tablet.cc:2369] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:24.085748 32674 tablet.cc:2369] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:24.131465 32540 tablet.cc:2369] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:24.240949 32698 raft_consensus.cc:482] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:24.241206 32698 raft_consensus.cc:504] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:24.242365 32698 leader_election.cc:288] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffdbddbf278140169abcbdc34f02def1 (127.14.209.131:40319), 11ab56a2129541a5881fbbe402940de4 (127.14.209.129:45581)
I20260620 06:15:24.246567 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.242652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:40319 (local address 127.14.209.130:49211)
0620 06:15:24.243147 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:24.243162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:24.243266 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:24.243617 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:24.243621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:24.243642 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:24.243782 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.243788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.244603 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.244609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:24.245506 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.245515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.246173 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.246177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:24.246285 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.246302 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:15:24.246362 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 06:15:24.246411 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":39,"spinlock_wait_cycles":3328,"thread_start_us":76,"threads_started":1}
I20260620 06:15:24.246641 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.242749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:49211 (local address 127.14.209.131:40319)
0620 06:15:24.242895 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:24.242900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:24.243230 (+ 330us) server_negotiation.cc:407] Connection header received
0620 06:15:24.243427 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:24.243433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:24.243499 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:24.243600 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:24.243901 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.244493 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.245659 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.246037 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.246169 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.246229 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:24.246413 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 06:15:24.246517 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:15:24.246802 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.242642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45581 (local address 127.14.209.130:34035)
0620 06:15:24.243019 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:24.243071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:24.243191 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:24.243647 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:24.243650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:24.243691 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:24.243824 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.243831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.244618 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.244620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:24.245496 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.245505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.246148 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.246152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:24.246257 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.246284 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:15:24.246563 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 06:15:24.246642 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":22,"spinlock_wait_cycles":1408,"thread_start_us":91,"threads_started":1}
I20260620 06:15:24.246907 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.242759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:34035 (local address 127.14.209.129:45581)
0620 06:15:24.242895 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:24.242900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:24.243158 (+ 258us) server_negotiation.cc:407] Connection header received
0620 06:15:24.243345 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:24.243352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:24.243417 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:24.243490 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:24.243958 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.244504 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.245618 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.246023 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.246137 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.246191 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:24.246594 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 06:15:24.246695 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:15:24.247383 32627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5779b6ee2b49938194ab03be999cda" candidate_uuid: "45f0c433901449ada80a34d007ea279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffdbddbf278140169abcbdc34f02def1" is_pre_election: true
I20260620 06:15:24.247416 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5779b6ee2b49938194ab03be999cda" candidate_uuid: "45f0c433901449ada80a34d007ea279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ab56a2129541a5881fbbe402940de4" is_pre_election: true
I20260620 06:15:24.247681 32627 raft_consensus.cc:2454] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f0c433901449ada80a34d007ea279a in term 0.
I20260620 06:15:24.247681 32359 raft_consensus.cc:2454] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f0c433901449ada80a34d007ea279a in term 0.
I20260620 06:15:24.248217 32428 leader_election.cc:302] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11ab56a2129541a5881fbbe402940de4, 45f0c433901449ada80a34d007ea279a, ffdbddbf278140169abcbdc34f02def1; no voters:
I20260620 06:15:24.248493 32698 raft_consensus.cc:2781] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:15:24.248574 32698 raft_consensus.cc:482] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:15:24.248616 32698 raft_consensus.cc:3037] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:24.249609 32698 raft_consensus.cc:504] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:24.249940 32698 leader_election.cc:288] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [CANDIDATE]: Term 1 election: Requested vote from peers ffdbddbf278140169abcbdc34f02def1 (127.14.209.131:40319), 11ab56a2129541a5881fbbe402940de4 (127.14.209.129:45581)
I20260620 06:15:24.250305 32627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5779b6ee2b49938194ab03be999cda" candidate_uuid: "45f0c433901449ada80a34d007ea279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffdbddbf278140169abcbdc34f02def1"
I20260620 06:15:24.250344 32359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5779b6ee2b49938194ab03be999cda" candidate_uuid: "45f0c433901449ada80a34d007ea279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ab56a2129541a5881fbbe402940de4"
I20260620 06:15:24.250435 32627 raft_consensus.cc:3037] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:24.250445 32359 raft_consensus.cc:3037] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:24.251708 32627 raft_consensus.cc:2454] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f0c433901449ada80a34d007ea279a in term 1.
I20260620 06:15:24.251715 32359 raft_consensus.cc:2454] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f0c433901449ada80a34d007ea279a in term 1.
I20260620 06:15:24.252012 32426 leader_election.cc:302] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f0c433901449ada80a34d007ea279a, ffdbddbf278140169abcbdc34f02def1; no voters:
I20260620 06:15:24.252213 32698 raft_consensus.cc:2781] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:24.252406 32698 raft_consensus.cc:686] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 1 LEADER]: Becoming Leader. State: Replica: 45f0c433901449ada80a34d007ea279a, State: Running, Role: LEADER
I20260620 06:15:24.252707 32698 consensus_queue.cc:237] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } }
I20260620 06:15:24.255630 32212 catalog_manager.cc:5495] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a reported cstate change: term changed from 0 to 1, leader changed from <none> to 45f0c433901449ada80a34d007ea279a (127.14.209.130). New cstate: current_term: 1 leader_uuid: "45f0c433901449ada80a34d007ea279a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45f0c433901449ada80a34d007ea279a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46517 } health_report { overall_health: HEALTHY } } }
I20260620 06:15:24.339802 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.336651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.14.209.130:46517)
0620 06:15:24.336775 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:24.336778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:24.336792 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:15:24.336877 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:24.336879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:24.336933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:24.337027 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:15:24.337229 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.337719 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.338522 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.338742 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.338816 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.339068 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:15:24.339435 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:15:24.339520 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:24.339633 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:15:24.339680 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:15:24.345320 32627 raft_consensus.cc:1261] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 1 FOLLOWER]: Refusing update from remote peer 45f0c433901449ada80a34d007ea279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:24.345338 32359 raft_consensus.cc:1261] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 1 FOLLOWER]: Refusing update from remote peer 45f0c433901449ada80a34d007ea279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:24.345969 32698 consensus_queue.cc:1035] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ab56a2129541a5881fbbe402940de4" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 45581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:24.346185 32701 consensus_queue.cc:1035] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffdbddbf278140169abcbdc34f02def1" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:24.356815 32705 mvcc.cc:205] Tried to move back new op lower bound from 7298810365311471616 to 7298810364941570048. Current Snapshot: MvccSnapshot[applied={T|T < 7298810365311471616 or (T in {7298810365311471616})}]
I20260620 06:15:24.361492 32212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:24.361642 32212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:24.363848 32212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46fc0a47ef944eea9405d04a2c5818ac: Sending DeleteTablet for 3 replicas of tablet 9d5779b6ee2b49938194ab03be999cda
I20260620 06:15:24.364470 32607 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5779b6ee2b49938194ab03be999cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:54170
I20260620 06:15:24.364490 32339 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5779b6ee2b49938194ab03be999cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:59078
I20260620 06:15:24.364672 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5779b6ee2b49938194ab03be999cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:24 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:53982
I20260620 06:15:24.365031 32714 tablet_replica.cc:330] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: stopping tablet replica
I20260620 06:15:24.365047 32716 tablet_replica.cc:330] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: stopping tablet replica
I20260620 06:15:24.365275 32715 tablet_replica.cc:330] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4: stopping tablet replica
I20260620 06:15:24.365324 32716 raft_consensus.cc:2229] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:15:24.365626 32714 raft_consensus.cc:2229] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:24.365672 32716 raft_consensus.cc:2258] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:24.365854 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 32276
I20260620 06:15:24.366042 32715 raft_consensus.cc:2229] T 9d5779b6ee2b49938194ab03be999cda P 11ab56a2129541a5881fbbe402940de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:24.366518 32714 raft_consensus.cc:2258] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:24.369799 32714 ts_tablet_manager.cc:1905] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:24.369825 32716 ts_tablet_manager.cc:1905] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:24.372545 32716 ts_tablet_manager.cc:1918] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:24.372748 32716 log.cc:1196] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/9d5779b6ee2b49938194ab03be999cda
I20260620 06:15:24.373121 32714 ts_tablet_manager.cc:1918] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:24.373189 32716 ts_tablet_manager.cc:1939] T 9d5779b6ee2b49938194ab03be999cda P 45f0c433901449ada80a34d007ea279a: Deleting consensus metadata
I20260620 06:15:24.373294 32714 log.cc:1196] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/9d5779b6ee2b49938194ab03be999cda
W20260620 06:15:24.373368 32200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:45581 (error 108)
I20260620 06:15:24.373631 32714 ts_tablet_manager.cc:1939] T 9d5779b6ee2b49938194ab03be999cda P ffdbddbf278140169abcbdc34f02def1: Deleting consensus metadata
I20260620 06:15:24.374156 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 32410
I20260620 06:15:24.375802 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.375403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:45581 (local address 127.0.0.1:59092)
0620 06:15:24.375603 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:24.375704 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 06:15:24.375911 32198 catalog_manager.cc:4841] TS ffdbddbf278140169abcbdc34f02def1 (127.14.209.131:40319): tablet 9d5779b6ee2b49938194ab03be999cda (table test-table [id=657d26ee44d24841ae06103c483aab23]) successfully deleted
W20260620 06:15:24.376004 32200 catalog_manager.cc:4584] TS 11ab56a2129541a5881fbbe402940de4 (127.14.209.129:45581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d5779b6ee2b49938194ab03be999cda: Network error: Client connection negotiation failed: client connection to 127.14.209.129:45581: connect: Connection refused (error 111)
I20260620 06:15:24.380012 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 32544
I20260620 06:15:24.381191 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.380087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46517 (local address 127.0.0.1:54004)
0620 06:15:24.381024 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:24.381087 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.130:46517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":847}
I20260620 06:15:24.385358 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 32182
2026-06-20T06:15:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2564 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T06:15:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:24Z Disabled control of system clock
I20260620 06:15:24.414924 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40027
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:40027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:15:24.520570 32721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:24.520898 32721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:24.520937 32721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:24.524889 32721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:24.524935 32721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:24.524952 32721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:24.524968 32721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:24.524983 32721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:24.529558 32721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:40027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:24.530354 32721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:24.531558 32721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:24.539372 32727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:24.539378 32729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:24.539378 32726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:24.539428 32721 server_base.cc:1034] running on GCE node
I20260620 06:15:24.540108 32721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:24.541021 32721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:24.542178 32721 hybrid_clock.cc:648] HybridClock initialized: now 1781936124542153 us; error 39 us; skew 500 ppm
I20260620 06:15:24.543839 32721 webserver.cc:460] Webserver started at http://127.14.209.190:46613/ using document root <none> and password file <none>
I20260620 06:15:24.544320 32721 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:24.544378 32721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:24.544521 32721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:24.546283 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "1aea1a08aa5841309672d75cb90e43a6"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "87e593ce4934a6e39c0b08bcb444219a"
server_key_iv: "86d21aa400e32d54890c6a4a6b51785d"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.546751 32721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "1aea1a08aa5841309672d75cb90e43a6"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "87e593ce4934a6e39c0b08bcb444219a"
server_key_iv: "86d21aa400e32d54890c6a4a6b51785d"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.550081 32721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260620 06:15:24.552309 32735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.553414 32721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:24.553506 32721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "1aea1a08aa5841309672d75cb90e43a6"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "87e593ce4934a6e39c0b08bcb444219a"
server_key_iv: "86d21aa400e32d54890c6a4a6b51785d"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.553582 32721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:24.571524 32721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:24.572018 32721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:24.572144 32721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:24.578473 32721 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:40027
I20260620 06:15:24.578488 319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:40027 every 8 connection(s)
I20260620 06:15:24.579380 32721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:24.579658 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 32721
I20260620 06:15:24.579803 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:24.580111 15174 external_mini_cluster.cc:1428] Setting key adcfb9e4631e8cc9b62122969e6e0bb0
I20260620 06:15:24.582546 320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:24.587509 320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6: Bootstrap starting.
I20260620 06:15:24.588582 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.581350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.14.209.190:40027)
0620 06:15:24.581955 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:24.581969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:24.582000 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:15:24.582588 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:24.582603 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:24.583008 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:24.583458 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:24.583842 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.584795 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.585657 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.586098 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.586270 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.587035 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:24.587039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:24.587087 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:24.587450 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:24.587499 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:24.587517 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:24.587621 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:24.587740 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 06:15:24.587965 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":144,"threads_started":1}
I20260620 06:15:24.589656 320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:24.590480 320 log.cc:824] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:24.592125 320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6: No bootstrap required, opened a new log
I20260620 06:15:24.596002 320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aea1a08aa5841309672d75cb90e43a6" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40027 } }
I20260620 06:15:24.596257 320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:24.596320 320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aea1a08aa5841309672d75cb90e43a6, State: Initialized, Role: FOLLOWER
I20260620 06:15:24.596731 320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aea1a08aa5841309672d75cb90e43a6" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40027 } }
I20260620 06:15:24.596879 320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:24.596943 320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:24.597035 320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:24.597949 320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aea1a08aa5841309672d75cb90e43a6" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40027 } }
I20260620 06:15:24.598282 320 leader_election.cc:302] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aea1a08aa5841309672d75cb90e43a6; no voters:
I20260620 06:15:24.598556 320 leader_election.cc:288] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:24.599646 320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:24.600930 326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:24.601202 326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [term 1 LEADER]: Becoming Leader. State: Replica: 1aea1a08aa5841309672d75cb90e43a6, State: Running, Role: LEADER
I20260620 06:15:24.601498 326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aea1a08aa5841309672d75cb90e43a6" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40027 } }
I20260620 06:15:24.603260 325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aea1a08aa5841309672d75cb90e43a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aea1a08aa5841309672d75cb90e43a6" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40027 } } }
I20260620 06:15:24.603309 333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aea1a08aa5841309672d75cb90e43a6. Latest consensus state: current_term: 1 leader_uuid: "1aea1a08aa5841309672d75cb90e43a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aea1a08aa5841309672d75cb90e43a6" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 40027 } } }
I20260620 06:15:24.603372 325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:24.603415 333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:24.603729 340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:24.606134 340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:24.610597 340 catalog_manager.cc:1347] Generated new cluster ID: e21469b8264b45f2b220c4f814ae9b8d
I20260620 06:15:24.610658 340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:24.616515 340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:24.617336 340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:24.628412 340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6: Generated new TSK 0
I20260620 06:15:24.629022 340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:24.631826 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40027
--builtin_ntp_servers=127.14.209.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:15:24.737300 344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:24.737509 344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:24.737543 344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:24.741460 344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:24.741504 344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:24.741595 344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:24.746162 344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:24.747044 344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:24.748286 344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:24.757187 352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:24.757194 350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:24.757194 349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:24.757319 344 server_base.cc:1034] running on GCE node
I20260620 06:15:24.757859 344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:24.758436 344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:24.759604 344 hybrid_clock.cc:648] HybridClock initialized: now 1781936124759573 us; error 49 us; skew 500 ppm
I20260620 06:15:24.761201 344 webserver.cc:460] Webserver started at http://127.14.209.129:44329/ using document root <none> and password file <none>
I20260620 06:15:24.761726 344 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:24.761811 344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:24.762039 344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:24.763901 344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "0aedeba8302c4c1eaecc384460a41edf"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "1479c5ab8d9fe6d86088de0b63424599"
server_key_iv: "c897fc98798440f6452c11f31ea38109"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.764416 344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "0aedeba8302c4c1eaecc384460a41edf"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "1479c5ab8d9fe6d86088de0b63424599"
server_key_iv: "c897fc98798440f6452c11f31ea38109"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.767616 344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:24.769855 358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.770902 344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:24.771020 344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "0aedeba8302c4c1eaecc384460a41edf"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "1479c5ab8d9fe6d86088de0b63424599"
server_key_iv: "c897fc98798440f6452c11f31ea38109"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.771164 344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:24.792294 344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:24.792848 344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:24.793020 344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:24.793550 344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:24.794374 344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:24.794440 344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.794499 344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:24.794544 344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.803517 344 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:46793
I20260620 06:15:24.803572 471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:46793 every 8 connection(s)
I20260620 06:15:24.804567 344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:24.807799 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 344
I20260620 06:15:24.807921 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:15:24.808192 15174 external_mini_cluster.cc:1428] Setting key 3e53ef81a7b5ccf24aa2f42149686fb3
I20260620 06:15:24.810065 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40027
--builtin_ntp_servers=127.14.209.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:15:24.813932 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.806316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:53519 (local address 127.14.209.190:40027)
0620 06:15:24.806467 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:24.806471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:24.807532 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 06:15:24.809265 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:24.809268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:24.809312 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:24.809413 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:24.810368 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.810891 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.812050 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.812280 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.812389 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.813455 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:24.813458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:24.813461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:24.813473 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:24.813491 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:24.813500 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:24.813575 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:24.813724 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 06:15:24.813795 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:15:24.814679 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.806701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40027 (local address 127.14.209.129:53519)
0620 06:15:24.807404 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:24.807441 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:24.809046 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:24.809591 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:24.809599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:24.809706 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:24.810217 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.810239 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.811018 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.811021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:24.811925 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.811935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.812422 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.812426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:24.812582 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:24.813213 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:24.813215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:24.813250 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:24.813308 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:24.813589 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:24.813603 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:24.813609 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:15:24.813723 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:15:24.813988 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260620 06:15:24.815834 472 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40027
I20260620 06:15:24.816045 472 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:24.816515 472 heartbeater.cc:511] Master 127.14.209.190:40027 requested a full tablet report, sending...
I20260620 06:15:24.818074 32752 ts_manager.cc:194] Registered new tserver with Master: 0aedeba8302c4c1eaecc384460a41edf (127.14.209.129:46793)
I20260620 06:15:24.819813 32752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.129:53519
W20260620 06:15:24.914800 475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:24.915016 475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:24.915091 475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:24.918882 475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:24.918926 475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:24.918977 475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:24.923450 475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:24.924310 475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:24.925522 475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:24.934235 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:24.934252 475 server_base.cc:1034] running on GCE node
W20260620 06:15:24.934283 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:24.934273 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:24.934851 475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:24.935431 475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:24.936604 475 hybrid_clock.cc:648] HybridClock initialized: now 1781936124936588 us; error 30 us; skew 500 ppm
I20260620 06:15:24.938171 475 webserver.cc:460] Webserver started at http://127.14.209.130:35405/ using document root <none> and password file <none>
I20260620 06:15:24.938686 475 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:24.938767 475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:24.938966 475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:24.940775 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "ff35ea7f0db340f3b3fed657cd7c5d40"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "f8aa0158232ac5bd07cee249d2b17ca4"
server_key_iv: "f7bcf419658e954135aa974848e72588"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.941282 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "ff35ea7f0db340f3b3fed657cd7c5d40"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "f8aa0158232ac5bd07cee249d2b17ca4"
server_key_iv: "f7bcf419658e954135aa974848e72588"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.944566 475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:24.946766 489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.947896 475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:24.948014 475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "ff35ea7f0db340f3b3fed657cd7c5d40"
format_stamp: "Formatted at 2026-06-20 06:15:24 on dist-test-slave-zt01"
server_key: "f8aa0158232ac5bd07cee249d2b17ca4"
server_key_iv: "f7bcf419658e954135aa974848e72588"
server_key_version: "encryptionkey@0"
I20260620 06:15:24.948108 475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:24.981915 475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:24.982478 475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:24.982658 475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:24.983187 475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:24.984005 475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:24.984068 475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.984135 475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:24.984175 475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:24.992722 475 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:45443
I20260620 06:15:24.992762 602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:45443 every 8 connection(s)
I20260620 06:15:24.993477 475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:24.995656 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 475
I20260620 06:15:24.995774 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:15:24.996068 15174 external_mini_cluster.cc:1428] Setting key d2802b720900ef972de4c863f89b568e
I20260620 06:15:24.997910 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40027
--builtin_ntp_servers=127.14.209.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:15:25.003276 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.995030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:44029 (local address 127.14.209.190:40027)
0620 06:15:24.995174 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:24.995178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:24.996268 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 06:15:24.997777 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:24.997780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:24.997833 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:24.997913 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:24.998778 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.999638 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.000940 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.001358 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.001477 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.002664 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:25.002671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:25.002676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:25.002701 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:25.002724 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:25.002738 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:25.002821 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.002977 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.003072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:15:25.004042 605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:24.995374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40027 (local address 127.14.209.130:44029)
0620 06:15:24.996094 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:24.996156 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:24.997596 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:24.998093 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:24.998103 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:24.998203 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:24.998647 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:24.998659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:24.999776 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:24.999779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.000799 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.000809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.001524 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.001530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.001718 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.002413 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:25.002415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:25.002451 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:25.002507 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:25.002838 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:25.002846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:25.002850 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.002941 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.003202 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":150,"threads_started":1}
I20260620 06:15:25.005198 603 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40027
I20260620 06:15:25.005441 603 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:25.006050 603 heartbeater.cc:511] Master 127.14.209.190:40027 requested a full tablet report, sending...
I20260620 06:15:25.007206 32752 ts_manager.cc:194] Registered new tserver with Master: ff35ea7f0db340f3b3fed657cd7c5d40 (127.14.209.130:45443)
I20260620 06:15:25.007781 32752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.130:44029
W20260620 06:15:25.105551 606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:25.105818 606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:25.105897 606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:25.109951 606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:25.110018 606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:25.110100 606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:25.114718 606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:25.115703 606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:25.116933 606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:25.125931 612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:25.125931 611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:25.125928 614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:25.126047 606 server_base.cc:1034] running on GCE node
I20260620 06:15:25.126559 606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:25.127188 606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:25.128357 606 hybrid_clock.cc:648] HybridClock initialized: now 1781936125128348 us; error 29 us; skew 500 ppm
I20260620 06:15:25.129954 606 webserver.cc:460] Webserver started at http://127.14.209.131:35617/ using document root <none> and password file <none>
I20260620 06:15:25.130474 606 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:25.130530 606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:25.130717 606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:25.132604 606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "e1bd7d84c3654c6c9ac26f26d3592206"
format_stamp: "Formatted at 2026-06-20 06:15:25 on dist-test-slave-zt01"
server_key: "e7f1117316a69c5f23799a0287e13cd1"
server_key_iv: "95bf127013a906b79f3f4851054e9b5b"
server_key_version: "encryptionkey@0"
I20260620 06:15:25.133118 606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "e1bd7d84c3654c6c9ac26f26d3592206"
format_stamp: "Formatted at 2026-06-20 06:15:25 on dist-test-slave-zt01"
server_key: "e7f1117316a69c5f23799a0287e13cd1"
server_key_iv: "95bf127013a906b79f3f4851054e9b5b"
server_key_version: "encryptionkey@0"
I20260620 06:15:25.136480 606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 06:15:25.138586 620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:25.139648 606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:25.139775 606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "e1bd7d84c3654c6c9ac26f26d3592206"
format_stamp: "Formatted at 2026-06-20 06:15:25 on dist-test-slave-zt01"
server_key: "e7f1117316a69c5f23799a0287e13cd1"
server_key_iv: "95bf127013a906b79f3f4851054e9b5b"
server_key_version: "encryptionkey@0"
I20260620 06:15:25.139878 606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:25.151976 606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:25.152513 606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:25.152688 606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:25.153185 606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:25.154016 606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:25.154083 606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:25.154138 606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:25.154176 606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:25.162631 606 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:43721
I20260620 06:15:25.162709 733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:43721 every 8 connection(s)
I20260620 06:15:25.163477 606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:25.163791 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 606
I20260620 06:15:25.163895 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:15:25.164191 15174 external_mini_cluster.cc:1428] Setting key cddb3b593c8cb6750953b028adcb16fb
I20260620 06:15:25.172336 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.165315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:44755 (local address 127.14.209.190:40027)
0620 06:15:25.165437 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.165441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.166678 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 06:15:25.167988 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.167991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.168036 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.168104 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:25.168834 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.169337 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.170438 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.170751 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.170834 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.171882 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:25.171885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:25.171888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:25.171899 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:25.171916 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:25.171923 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:25.171988 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.172176 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.172227 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:15:25.173067 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.165806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:40027 (local address 127.14.209.131:44755)
0620 06:15:25.166521 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.166557 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:25.167824 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:25.168214 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:25.168220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:25.168292 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:25.168717 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.168728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.169458 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.169462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.170306 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.170316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.170873 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.170876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.171012 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.171635 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:25.171637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:25.171671 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:25.171726 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:25.172034 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:25.172042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:25.172047 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.172160 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.172357 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260620 06:15:25.174165 734 heartbeater.cc:347] Connected to a master server at 127.14.209.190:40027
I20260620 06:15:25.174398 734 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:25.174894 734 heartbeater.cc:511] Master 127.14.209.190:40027 requested a full tablet report, sending...
I20260620 06:15:25.175913 32752 ts_manager.cc:194] Registered new tserver with Master: e1bd7d84c3654c6c9ac26f26d3592206 (127.14.209.131:43721)
I20260620 06:15:25.176499 32752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.131:44755
I20260620 06:15:25.177968 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:15:25.183274 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.179989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.14.209.190:40027)
0620 06:15:25.180090 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.180093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.180266 (+ 173us) server_negotiation.cc:407] Connection header received
0620 06:15:25.180353 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.180355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.180394 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.180456 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:25.180640 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.181106 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.181869 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.182062 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.182133 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.182806 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:25.182809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:25.182811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:25.182822 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:25.182839 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:25.182847 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:25.182912 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.182999 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.183039 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:15:25.185545 32752 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:15:25.188478 32752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:15:25.202596 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.197520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.14.209.131:43721)
0620 06:15:25.197977 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.197992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.198018 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:15:25.198183 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.198188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.198381 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.198525 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:25.198992 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.199765 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.201326 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.201977 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.202105 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.202263 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.202359 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.202442 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":113,"threads_started":1}
I20260620 06:15:25.202595 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.197103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:43721 (local address 127.0.0.1:35404)
0620 06:15:25.197954 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.197966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:25.198045 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:25.198655 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:25.198657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:25.198669 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:25.198871 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.198878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.199912 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.199914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.201189 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.201200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.202092 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.202096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.202191 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.202236 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.202336 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.202442 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":57,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260620 06:15:25.203342 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.196808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46793 (local address 127.0.0.1:38780)
0620 06:15:25.197557 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.197603 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:25.197764 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:25.198765 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:25.198767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:25.198778 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:25.198928 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.198937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.200414 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.200417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.201527 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.201536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.202752 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.202755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.202947 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.203033 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.203124 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.203211 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":119,"spinlock_wait_cycles":4864,"thread_start_us":79,"threads_started":1}
I20260620 06:15:25.203415 746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.196940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.14.209.129:46793)
0620 06:15:25.197378 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.197390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.197890 (+ 500us) server_negotiation.cc:407] Connection header received
0620 06:15:25.197973 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.197978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.198476 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.198621 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:25.199085 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.200294 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.201653 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.202410 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.202532 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.202698 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.203164 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.203265 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":115,"threads_started":1}
I20260620 06:15:25.203480 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.197555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.14.209.130:45443)
0620 06:15:25.198199 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.198211 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.198251 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:15:25.198324 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.198328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.198471 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.198621 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:25.199012 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.200259 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.201697 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.202417 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.202532 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.202683 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.203208 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.203302 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":134,"threads_started":1}
I20260620 06:15:25.203764 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.196999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:45443 (local address 127.0.0.1:60570)
0620 06:15:25.197557 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.197604 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:25.197750 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:25.198619 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:25.198635 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:25.198659 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:25.198862 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.198869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.200369 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.200372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.201585 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.201594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.202752 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.202755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.202950 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.203033 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.203124 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.203638 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":567,"spinlock_wait_cycles":9728,"thread_start_us":90,"threads_started":1}
I20260620 06:15:25.205751 668 tablet_service.cc:1467] Processing CreateTablet for tablet 674ff2e23bbc46aea9359c7baa68db7c (DEFAULT_TABLE table=test-table [id=318b23a5be7d4011a1e6e012cae48170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:25.206205 537 tablet_service.cc:1467] Processing CreateTablet for tablet 674ff2e23bbc46aea9359c7baa68db7c (DEFAULT_TABLE table=test-table [id=318b23a5be7d4011a1e6e012cae48170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:25.206833 668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ff2e23bbc46aea9359c7baa68db7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:25.206528 406 tablet_service.cc:1467] Processing CreateTablet for tablet 674ff2e23bbc46aea9359c7baa68db7c (DEFAULT_TABLE table=test-table [id=318b23a5be7d4011a1e6e012cae48170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:25.207109 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ff2e23bbc46aea9359c7baa68db7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:25.207453 406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ff2e23bbc46aea9359c7baa68db7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:25.212672 751 tablet_bootstrap.cc:492] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: Bootstrap starting.
I20260620 06:15:25.214051 752 tablet_bootstrap.cc:492] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Bootstrap starting.
I20260620 06:15:25.214051 753 tablet_bootstrap.cc:492] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: Bootstrap starting.
I20260620 06:15:25.215477 751 tablet_bootstrap.cc:654] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:25.216006 753 tablet_bootstrap.cc:654] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:25.216006 752 tablet_bootstrap.cc:654] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:25.216215 751 log.cc:824] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:25.216768 753 log.cc:824] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:25.216769 752 log.cc:824] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:25.217927 751 tablet_bootstrap.cc:492] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: No bootstrap required, opened a new log
I20260620 06:15:25.218107 751 ts_tablet_manager.cc:1397] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:15:25.218380 752 tablet_bootstrap.cc:492] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: No bootstrap required, opened a new log
I20260620 06:15:25.218559 752 ts_tablet_manager.cc:1397] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:15:25.220525 753 tablet_bootstrap.cc:492] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: No bootstrap required, opened a new log
I20260620 06:15:25.220696 753 ts_tablet_manager.cc:1397] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 06:15:25.221513 751 raft_consensus.cc:348] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.221735 751 raft_consensus.cc:374] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:25.221812 751 raft_consensus.cc:729] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff35ea7f0db340f3b3fed657cd7c5d40, State: Initialized, Role: FOLLOWER
I20260620 06:15:25.222239 751 consensus_queue.cc:260] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.222455 752 raft_consensus.cc:348] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.222755 752 raft_consensus.cc:374] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:25.222827 603 heartbeater.cc:503] Master 127.14.209.190:40027 was elected leader, sending a full tablet report...
I20260620 06:15:25.222854 752 raft_consensus.cc:729] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bd7d84c3654c6c9ac26f26d3592206, State: Initialized, Role: FOLLOWER
I20260620 06:15:25.223177 751 ts_tablet_manager.cc:1428] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:15:25.223305 752 consensus_queue.cc:260] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.224094 734 heartbeater.cc:503] Master 127.14.209.190:40027 was elected leader, sending a full tablet report...
I20260620 06:15:25.224248 752 ts_tablet_manager.cc:1428] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:15:25.224608 753 raft_consensus.cc:348] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.224807 753 raft_consensus.cc:374] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:25.224864 753 raft_consensus.cc:729] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aedeba8302c4c1eaecc384460a41edf, State: Initialized, Role: FOLLOWER
I20260620 06:15:25.225284 753 consensus_queue.cc:260] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.226133 753 ts_tablet_manager.cc:1428] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:15:25.226455 472 heartbeater.cc:503] Master 127.14.209.190:40027 was elected leader, sending a full tablet report...
W20260620 06:15:25.244266 604 tablet.cc:2369] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:25.305704 473 tablet.cc:2369] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:25.414458 735 tablet.cc:2369] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:25.519171 758 raft_consensus.cc:482] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:25.519379 758 raft_consensus.cc:504] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.520553 758 leader_election.cc:288] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0aedeba8302c4c1eaecc384460a41edf (127.14.209.129:46793), ff35ea7f0db340f3b3fed657cd7c5d40 (127.14.209.130:45443)
I20260620 06:15:25.524569 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.520890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:41413 (local address 127.14.209.130:45443)
0620 06:15:25.521014 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.521018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.521283 (+ 265us) server_negotiation.cc:407] Connection header received
0620 06:15:25.521378 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.521381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.521439 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.521517 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:25.521772 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.522495 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.523630 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.524026 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.524149 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.524197 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.524371 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.524437 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:15:25.524597 746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.520854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:59297 (local address 127.14.209.129:46793)
0620 06:15:25.520982 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.520989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.521006 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:15:25.521179 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.521182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.521235 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.521346 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:25.521649 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.522436 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.523671 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.524053 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.524193 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.524241 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.524399 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.524471 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:15:25.524597 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.520746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46793 (local address 127.14.209.131:59297)
0620 06:15:25.520939 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.520954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:25.521046 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:25.521357 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:25.521360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:25.521386 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:25.521534 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.521540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.522591 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.522595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.523539 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.523549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.524179 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.524183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.524280 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.524303 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.524385 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.524460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":21}
I20260620 06:15:25.524616 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.520832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:45443 (local address 127.14.209.131:41413)
0620 06:15:25.521196 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.521206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:25.521283 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:25.521532 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:25.521535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:25.521548 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:25.521678 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.521684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.522609 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.522613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.523510 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:25.523519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.524143 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.524147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:25.524242 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.524266 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:15:25.524356 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:15:25.524431 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260620 06:15:25.525182 557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ff2e23bbc46aea9359c7baa68db7c" candidate_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" is_pre_election: true
I20260620 06:15:25.525194 426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ff2e23bbc46aea9359c7baa68db7c" candidate_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aedeba8302c4c1eaecc384460a41edf" is_pre_election: true
I20260620 06:15:25.525449 557 raft_consensus.cc:2454] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bd7d84c3654c6c9ac26f26d3592206 in term 0.
I20260620 06:15:25.525470 426 raft_consensus.cc:2454] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bd7d84c3654c6c9ac26f26d3592206 in term 0.
I20260620 06:15:25.525926 621 leader_election.cc:302] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0aedeba8302c4c1eaecc384460a41edf, e1bd7d84c3654c6c9ac26f26d3592206, ff35ea7f0db340f3b3fed657cd7c5d40; no voters:
I20260620 06:15:25.526180 758 raft_consensus.cc:2781] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:15:25.526252 758 raft_consensus.cc:482] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:15:25.526299 758 raft_consensus.cc:3037] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:25.527364 758 raft_consensus.cc:504] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.527719 758 leader_election.cc:288] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [CANDIDATE]: Term 1 election: Requested vote from peers 0aedeba8302c4c1eaecc384460a41edf (127.14.209.129:46793), ff35ea7f0db340f3b3fed657cd7c5d40 (127.14.209.130:45443)
I20260620 06:15:25.527985 426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ff2e23bbc46aea9359c7baa68db7c" candidate_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aedeba8302c4c1eaecc384460a41edf"
I20260620 06:15:25.528051 557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ff2e23bbc46aea9359c7baa68db7c" candidate_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40"
I20260620 06:15:25.528110 426 raft_consensus.cc:3037] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:25.528175 557 raft_consensus.cc:3037] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:25.529083 426 raft_consensus.cc:2454] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bd7d84c3654c6c9ac26f26d3592206 in term 1.
I20260620 06:15:25.529084 557 raft_consensus.cc:2454] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bd7d84c3654c6c9ac26f26d3592206 in term 1.
I20260620 06:15:25.529379 621 leader_election.cc:302] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bd7d84c3654c6c9ac26f26d3592206, ff35ea7f0db340f3b3fed657cd7c5d40; no voters:
I20260620 06:15:25.529563 758 raft_consensus.cc:2781] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:25.529769 758 raft_consensus.cc:686] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 1 LEADER]: Becoming Leader. State: Replica: e1bd7d84c3654c6c9ac26f26d3592206, State: Running, Role: LEADER
I20260620 06:15:25.530062 758 consensus_queue.cc:237] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } }
I20260620 06:15:25.533077 32752 catalog_manager.cc:5495] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bd7d84c3654c6c9ac26f26d3592206 (127.14.209.131). New cstate: current_term: 1 leader_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bd7d84c3654c6c9ac26f26d3592206" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260620 06:15:25.560914 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.557538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.14.209.131:43721)
0620 06:15:25.557663 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:25.557666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:25.557679 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:15:25.557742 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:25.557744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:25.557793 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:25.557875 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:15:25.558079 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.558567 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.559491 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:25.559710 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:25.559807 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:25.560061 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:15:25.560439 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:15:25.560528 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:25.560661 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:15:25.560743 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:15:25.566350 426 raft_consensus.cc:1261] T 674ff2e23bbc46aea9359c7baa68db7c P 0aedeba8302c4c1eaecc384460a41edf [term 1 FOLLOWER]: Refusing update from remote peer e1bd7d84c3654c6c9ac26f26d3592206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:25.566350 557 raft_consensus.cc:1261] T 674ff2e23bbc46aea9359c7baa68db7c P ff35ea7f0db340f3b3fed657cd7c5d40 [term 1 FOLLOWER]: Refusing update from remote peer e1bd7d84c3654c6c9ac26f26d3592206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:25.567245 758 consensus_queue.cc:1035] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff35ea7f0db340f3b3fed657cd7c5d40" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:25.567593 761 consensus_queue.cc:1035] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aedeba8302c4c1eaecc384460a41edf" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:25.574177 764 mvcc.cc:205] Tried to move back new op lower bound from 7298810370312323072 to 7298810370173419520. Current Snapshot: MvccSnapshot[applied={T|T < 7298810370312323072}]
I20260620 06:15:25.574874 765 mvcc.cc:205] Tried to move back new op lower bound from 7298810370312323072 to 7298810370173419520. Current Snapshot: MvccSnapshot[applied={T|T < 7298810370312323072}]
I20260620 06:15:25.583205 32752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:25.583343 32752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:25.585681 32752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1aea1a08aa5841309672d75cb90e43a6: Sending DeleteTablet for 3 replicas of tablet 674ff2e23bbc46aea9359c7baa68db7c
I20260620 06:15:25.586374 668 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ff2e23bbc46aea9359c7baa68db7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:25 UTC) from {username='slave'} at 127.0.0.1:35404
I20260620 06:15:25.586377 537 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ff2e23bbc46aea9359c7baa68db7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:25 UTC) from {username='slave'} at 127.0.0.1:60570
I20260620 06:15:25.586800 406 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ff2e23bbc46aea9359c7baa68db7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:25 UTC) from {username='slave'} at 127.0.0.1:38780
I20260620 06:15:25.587035 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 344
W20260620 06:15:25.596268 32738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:46793 (error 108)
I20260620 06:15:25.596545 775 tablet_replica.cc:330] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: stopping tablet replica
I20260620 06:15:25.596837 775 raft_consensus.cc:2229] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:15:25.597134 775 raft_consensus.cc:2258] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:25.597407 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 475
I20260620 06:15:25.597474 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.597213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:46793 (local address 127.0.0.1:38788)
0620 06:15:25.597322 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.597389 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:46793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 06:15:25.598474 775 ts_tablet_manager.cc:1905] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:15:25.600627 32738 catalog_manager.cc:4584] TS 0aedeba8302c4c1eaecc384460a41edf (127.14.209.129:46793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 674ff2e23bbc46aea9359c7baa68db7c: Network error: Client connection negotiation failed: client connection to 127.14.209.129:46793: connect: Connection refused (error 111)
I20260620 06:15:25.601188 775 ts_tablet_manager.cc:1918] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:25.601275 775 log.cc:1196] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/674ff2e23bbc46aea9359c7baa68db7c
I20260620 06:15:25.601572 775 ts_tablet_manager.cc:1939] T 674ff2e23bbc46aea9359c7baa68db7c P e1bd7d84c3654c6c9ac26f26d3592206: Deleting consensus metadata
I20260620 06:15:25.602586 32739 catalog_manager.cc:4841] TS e1bd7d84c3654c6c9ac26f26d3592206 (127.14.209.131:43721): tablet 674ff2e23bbc46aea9359c7baa68db7c (table test-table [id=318b23a5be7d4011a1e6e012cae48170]) successfully deleted
I20260620 06:15:25.607862 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 606
I20260620 06:15:25.608069 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:25.607837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:45443 (local address 127.0.0.1:60580)
0620 06:15:25.607961 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:25.608009 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.130:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 06:15:25.612995 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 32721
2026-06-20T06:15:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1226 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:25 dist-test-slave-zt01 krb5kdc[780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:25 dist-test-slave-zt01 krb5kdc[780](info): set up 2 sockets
Jun 20 06:15:25 dist-test-slave-zt01 krb5kdc[780](info): commencing operation
krb5kdc: starting...
W20260620 06:15:26.650766 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:26 dist-test-slave-zt01 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:26.763804 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:38941
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:38941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:26.868033 793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:26.868283 793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:26.868340 793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:26.872244 793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:26.872292 793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:26.872311 793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:26.872327 793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:26.872381 793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:26.877497 793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:38941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:38941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:26.878419 793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:26.879323 793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:26.887259 798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:26.887358 793 server_base.cc:1034] running on GCE node
W20260620 06:15:26.887238 801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:26.887272 799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:26.888005 793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:26.888935 793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:26.890084 793 hybrid_clock.cc:648] HybridClock initialized: now 1781936126890067 us; error 29 us; skew 500 ppm
Jun 20 06:15:26 dist-test-slave-zt01 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936126, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:26.892645 793 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:26.893416 793 webserver.cc:460] Webserver started at http://127.14.209.190:34267/ using document root <none> and password file <none>
I20260620 06:15:26.893923 793 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:26.894006 793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:26.894212 793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:26.896126 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "2defda74b6174d81a6d59a0ba24ac461"
format_stamp: "Formatted at 2026-06-20 06:15:26 on dist-test-slave-zt01"
server_key: "d4cf22db63e742a0e5bdc86e95f73981"
server_key_iv: "ed0b26b87adad51f33f94d418fda8bdd"
server_key_version: "encryptionkey@0"
I20260620 06:15:26.896642 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "2defda74b6174d81a6d59a0ba24ac461"
format_stamp: "Formatted at 2026-06-20 06:15:26 on dist-test-slave-zt01"
server_key: "d4cf22db63e742a0e5bdc86e95f73981"
server_key_iv: "ed0b26b87adad51f33f94d418fda8bdd"
server_key_version: "encryptionkey@0"
I20260620 06:15:26.900200 793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:15:26.902578 808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:26.903764 793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:26.903892 793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "2defda74b6174d81a6d59a0ba24ac461"
format_stamp: "Formatted at 2026-06-20 06:15:26 on dist-test-slave-zt01"
server_key: "d4cf22db63e742a0e5bdc86e95f73981"
server_key_iv: "ed0b26b87adad51f33f94d418fda8bdd"
server_key_version: "encryptionkey@0"
I20260620 06:15:26.904000 793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:26.923202 793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:26.926081 793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:26.926259 793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:26.932608 793 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:38941
I20260620 06:15:26.932641 860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:38941 every 8 connection(s)
I20260620 06:15:26.933475 793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:26.936080 861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:26.939174 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 793
I20260620 06:15:26.939323 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:26.939613 15174 external_mini_cluster.cc:1428] Setting key fee508f149cd688acf97e244bfdd13ab
I20260620 06:15:26.941603 861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: Bootstrap starting.
I20260620 06:15:26.944078 861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:26.944876 861 log.cc:824] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:26.946785 861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: No bootstrap required, opened a new log
Jun 20 06:15:26 dist-test-slave-zt01 krb5kdc[780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:26.950525 861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2defda74b6174d81a6d59a0ba24ac461" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38941 } }
I20260620 06:15:26.950728 861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:26.950784 861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2defda74b6174d81a6d59a0ba24ac461, State: Initialized, Role: FOLLOWER
I20260620 06:15:26.951267 861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2defda74b6174d81a6d59a0ba24ac461" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38941 } }
I20260620 06:15:26.951418 861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:26.951500 861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:26.951594 861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:26.952487 861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2defda74b6174d81a6d59a0ba24ac461" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38941 } }
I20260620 06:15:26.952824 861 leader_election.cc:302] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2defda74b6174d81a6d59a0ba24ac461; no voters:
I20260620 06:15:26.953127 861 leader_election.cc:288] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:26.953190 866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:26.953395 866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [term 1 LEADER]: Becoming Leader. State: Replica: 2defda74b6174d81a6d59a0ba24ac461, State: Running, Role: LEADER
I20260620 06:15:26.953716 866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2defda74b6174d81a6d59a0ba24ac461" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38941 } }
I20260620 06:15:26.953850 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:26.940824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.14.209.190:38941)
0620 06:15:26.941184 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:26.941200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:26.941239 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:26.942032 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:26.942051 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:26.942424 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:26.942764 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:26.943419 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:26.944873 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:26.945730 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:26.946397 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:26.946553 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:26.949117 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:26.949122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:26.949133 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:26.949157 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:26.950848 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:26.951311 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:26.951315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:26.951318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:26.951457 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:26.951801 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:26.951804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:26.951806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:26.952169 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:26.952370 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:26.952656 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 06:15:26.952957 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260620 06:15:26.954314 861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:26.955235 868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2defda74b6174d81a6d59a0ba24ac461. Latest consensus state: current_term: 1 leader_uuid: "2defda74b6174d81a6d59a0ba24ac461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2defda74b6174d81a6d59a0ba24ac461" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38941 } } }
I20260620 06:15:26.955379 868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:26.955715 867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2defda74b6174d81a6d59a0ba24ac461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2defda74b6174d81a6d59a0ba24ac461" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 38941 } } }
I20260620 06:15:26.955792 867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461 [sys.catalog]: This master's current role is: LEADER
W20260620 06:15:26.958871 882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:26.958953 882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:26.959033 875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:26.959695 875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:26.964618 875 catalog_manager.cc:1347] Generated new cluster ID: 32e04e8fb2b4451880b1b840c62943bb
I20260620 06:15:26.964673 875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:26.991029 875 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:26.992188 875 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:27.000617 875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: Generated new TSK 0
I20260620 06:15:27.001318 875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:15:27.036106 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38941
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:27.139788 887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:27.139999 887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:27.140074 887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:27.144065 887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:27.144114 887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:27.144200 887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:27.149186 887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:27.150079 887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:27.150980 887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:27.162945 892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:27.162933 893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:27.163487 887 server_base.cc:1034] running on GCE node
W20260620 06:15:27.165371 895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:27.165855 887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:27.166594 887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:27.167869 887 hybrid_clock.cc:648] HybridClock initialized: now 1781936127167844 us; error 34 us; skew 500 ppm
Jun 20 06:15:27 dist-test-slave-zt01 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:27.171882 887 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:15:27.172936 887 webserver.cc:460] Webserver started at http://127.14.209.129:40851/ using document root <none> and password file <none>
I20260620 06:15:27.173544 887 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:27.173609 887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:27.173821 887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:27.176721 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "f3008595d95c4772bfb83eeeb0647740"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "9a770dcddb42a9a9743f38e85d7a7120"
server_key_iv: "01a2096f790489ee52ae4d520871e0f8"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.177459 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "f3008595d95c4772bfb83eeeb0647740"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "9a770dcddb42a9a9743f38e85d7a7120"
server_key_iv: "01a2096f790489ee52ae4d520871e0f8"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.182420 887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:15:27.186190 902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.187933 887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:27.188059 887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "f3008595d95c4772bfb83eeeb0647740"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "9a770dcddb42a9a9743f38e85d7a7120"
server_key_iv: "01a2096f790489ee52ae4d520871e0f8"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.188192 887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:27.202862 887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:27.205812 887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:27.205992 887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:27.206501 887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:27.207363 887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:27.207427 887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.207491 887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:27.207540 887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.216050 887 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:36829
I20260620 06:15:27.216091 1015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:36829 every 8 connection(s)
I20260620 06:15:27.216833 887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:27.221495 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 887
I20260620 06:15:27.221623 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:15:27.221884 15174 external_mini_cluster.cc:1428] Setting key b05d27e7f16883835e1512c277505b0a
Jun 20 06:15:27 dist-test-slave-zt01 krb5kdc[780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:27.229676 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.218353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:42271 (local address 127.14.209.190:38941)
0620 06:15:27.218475 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.218479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.219270 (+ 791us) server_negotiation.cc:407] Connection header received
0620 06:15:27.220507 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.220510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.220555 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.220644 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:27.221662 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.222197 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.223562 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.223792 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.223899 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.226336 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:27.226340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:27.226348 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:27.226369 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:27.227771 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.228278 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.228283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.228285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.228362 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.228680 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.228683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.228685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.228856 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:27.228957 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.229394 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.229509 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:15:27.230350 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.218689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38941 (local address 127.14.209.129:42271)
0620 06:15:27.219144 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.219183 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.220324 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.220777 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.220782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.221098 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:27.221506 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.221518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.222319 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.222324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.223442 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.223452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.223932 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.223936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.224093 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.224610 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:27.224625 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:27.226186 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:27.227920 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:27.227925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:27.227937 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:27.228180 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:27.228451 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:27.228454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:27.228456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:27.228583 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:27.228956 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:27.228962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:27.229180 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.229354 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.229630 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260620 06:15:27.231473 1016 heartbeater.cc:347] Connected to a master server at 127.14.209.190:38941
I20260620 06:15:27.231655 1016 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:27.232050 1016 heartbeater.cc:511] Master 127.14.209.190:38941 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
I20260620 06:15:27.233632 825 ts_manager.cc:194] Registered new tserver with Master: f3008595d95c4772bfb83eeeb0647740 (127.14.209.129:36829)
I20260620 06:15:27.234755 825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:42271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:15:27.248337 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38941
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:27.352756 1021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:27.352984 1021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:27.353049 1021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:27.357216 1021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:27.357281 1021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:27.357368 1021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:27.362288 1021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:27.363258 1021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:27.364120 1021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:27.372632 1026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:27.372720 1027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:27.372720 1029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:27.372882 1021 server_base.cc:1034] running on GCE node
I20260620 06:15:27.373234 1021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:27.373805 1021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:27.374980 1021 hybrid_clock.cc:648] HybridClock initialized: now 1781936127374960 us; error 34 us; skew 500 ppm
Jun 20 06:15:27 dist-test-slave-zt01 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:27.377600 1021 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:15:27.378352 1021 webserver.cc:460] Webserver started at http://127.14.209.130:40249/ using document root <none> and password file <none>
I20260620 06:15:27.378866 1021 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:27.378947 1021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:27.379189 1021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:27.380951 1021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "87c94f47578d4d599e0c25f71dfdcee5"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "964b01bb1d64688f49d4e8f9ac5336d9"
server_key_iv: "1b851fac7eb527cbd0dc9ac91c63f31e"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.381460 1021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "87c94f47578d4d599e0c25f71dfdcee5"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "964b01bb1d64688f49d4e8f9ac5336d9"
server_key_iv: "1b851fac7eb527cbd0dc9ac91c63f31e"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.384898 1021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:27.387154 1036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.388278 1021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:27.388409 1021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "87c94f47578d4d599e0c25f71dfdcee5"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "964b01bb1d64688f49d4e8f9ac5336d9"
server_key_iv: "1b851fac7eb527cbd0dc9ac91c63f31e"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.388509 1021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:27.413581 1021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:27.416556 1021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:27.416778 1021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:27.417308 1021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:27.418046 1021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:27.418111 1021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.418174 1021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:27.418218 1021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.426558 1021 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:32953
I20260620 06:15:27.426658 1149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:32953 every 8 connection(s)
I20260620 06:15:27.427435 1021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:27.433835 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1021
I20260620 06:15:27.433988 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:15:27.434298 15174 external_mini_cluster.cc:1428] Setting key bc612b91374e42a563fec2d386791cf3
Jun 20 06:15:27 dist-test-slave-zt01 krb5kdc[780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:27.441247 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.428996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:43703 (local address 127.14.209.190:38941)
0620 06:15:27.429109 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.429113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.429822 (+ 709us) server_negotiation.cc:407] Connection header received
0620 06:15:27.431147 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.431151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.431213 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.431304 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:27.432327 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.432851 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.433919 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.434194 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.434301 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.436668 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:27.436675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:27.436681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:27.436714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:27.438763 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.439497 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.439504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.439509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.439612 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.440015 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.440021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.440026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.440301 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:27.440414 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.440850 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.440991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:15:27.441754 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.429288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38941 (local address 127.14.209.130:43703)
0620 06:15:27.429684 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.429721 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.430917 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.431457 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.431464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.431771 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:27.432183 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.432195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.432969 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.432972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.433800 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.433809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.434332 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.434336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.434495 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.434981 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:27.435001 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:27.436502 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:27.439096 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:27.439105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:27.439121 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:27.439390 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:27.439736 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:27.439739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:27.439740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:27.439896 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:27.440405 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:27.440409 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:27.440608 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.440749 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.440987 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260620 06:15:27.442819 1150 heartbeater.cc:347] Connected to a master server at 127.14.209.190:38941
I20260620 06:15:27.443089 1150 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:27.443517 1150 heartbeater.cc:511] Master 127.14.209.190:38941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
I20260620 06:15:27.444464 825 ts_manager.cc:194] Registered new tserver with Master: 87c94f47578d4d599e0c25f71dfdcee5 (127.14.209.130:32953)
I20260620 06:15:27.445040 825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:43703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:15:27.463140 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38941
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:27.568513 1155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:27.568715 1155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:27.568753 1155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:27.572748 1155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:27.572795 1155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:27.572893 1155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:27.577991 1155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:38941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:27.578794 1155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:27.579684 1155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:27.589010 1160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:27.589003 1161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:27.589001 1163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:27.589221 1155 server_base.cc:1034] running on GCE node
I20260620 06:15:27.589653 1155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:27.590211 1155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:27.591357 1155 hybrid_clock.cc:648] HybridClock initialized: now 1781936127591333 us; error 38 us; skew 500 ppm
Jun 20 06:15:27 dist-test-slave-zt01 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:27.594255 1155 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:15:27.595023 1155 webserver.cc:460] Webserver started at http://127.14.209.131:32771/ using document root <none> and password file <none>
I20260620 06:15:27.595563 1155 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:27.595657 1155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:27.595862 1155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:27.597656 1155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "1ad5bd227192448e846c32e531fe9a5c"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "e71c99b909dfba1d97b1af8962a39d33"
server_key_iv: "dd9576ccdc67b55d982a8e3c2a31df31"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.598225 1155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "1ad5bd227192448e846c32e531fe9a5c"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "e71c99b909dfba1d97b1af8962a39d33"
server_key_iv: "dd9576ccdc67b55d982a8e3c2a31df31"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.601539 1155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:15:27.603706 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.604781 1155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:27.604902 1155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "1ad5bd227192448e846c32e531fe9a5c"
format_stamp: "Formatted at 2026-06-20 06:15:27 on dist-test-slave-zt01"
server_key: "e71c99b909dfba1d97b1af8962a39d33"
server_key_iv: "dd9576ccdc67b55d982a8e3c2a31df31"
server_key_version: "encryptionkey@0"
I20260620 06:15:27.604998 1155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:27.617798 1155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:27.620638 1155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:27.620817 1155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:27.621320 1155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:27.622035 1155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:27.622099 1155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.622159 1155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:27.622198 1155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:27.630762 1155 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:35543
I20260620 06:15:27.630782 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:35543 every 8 connection(s)
I20260620 06:15:27.631589 1155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:27.638827 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1155
I20260620 06:15:27.638980 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:15:27.639333 15174 external_mini_cluster.cc:1428] Setting key cd36b39323f59037bd9b85a34889b719
Jun 20 06:15:27 dist-test-slave-zt01 krb5kdc[780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936127, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:27.643841 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.633130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:48519 (local address 127.14.209.190:38941)
0620 06:15:27.633270 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.633274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.633983 (+ 709us) server_negotiation.cc:407] Connection header received
0620 06:15:27.635272 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.635275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.635327 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.635391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:27.636363 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.636865 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.637910 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.638112 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.638185 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.640697 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:27.640700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:27.640702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:27.640723 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:27.642107 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.642547 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.642549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.642550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.642616 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.642883 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.642885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.642886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.643037 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:27.643172 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.643630 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.643718 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:15:27.644578 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.633466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:38941 (local address 127.14.209.131:48519)
0620 06:15:27.633851 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.633887 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.635035 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.635522 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.635528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.635823 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:27.636227 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.636238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.636960 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.636962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.637810 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.637824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.638220 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.638222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.638355 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.638833 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:27.638857 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:27.640541 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:27.642203 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:27.642207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:27.642217 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:27.642465 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:27.642689 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:27.642690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:27.642692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:27.642807 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:27.643231 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:27.643236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:27.643447 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.643588 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.643795 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
I20260620 06:15:27.645550 1284 heartbeater.cc:347] Connected to a master server at 127.14.209.190:38941
I20260620 06:15:27.645745 1284 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:27.646144 1284 heartbeater.cc:511] Master 127.14.209.190:38941 requested a full tablet report, sending...
I20260620 06:15:27.646996 825 ts_manager.cc:194] Registered new tserver with Master: 1ad5bd227192448e846c32e531fe9a5c (127.14.209.131:35543)
I20260620 06:15:27.647529 825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:48519
I20260620 06:15:27.652557 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:15:27.661162 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.654304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.14.209.190:38941)
0620 06:15:27.654427 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.654430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.654606 (+ 176us) server_negotiation.cc:407] Connection header received
0620 06:15:27.654691 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.654693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.654725 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.654782 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:27.655533 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.656005 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.656973 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.657167 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.657235 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.658516 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:27.658519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:27.658521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:27.658541 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:27.659752 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.660182 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.660184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.660185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.660250 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:27.660514 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:27.660516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:27.660517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:27.660681 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:27.660785 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.660995 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.661062 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:15:27.663647 825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:15:27.666455 825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:15:27.679948 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.673637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:36829 (local address 127.0.0.1:40040)
0620 06:15:27.674300 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.674352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.674468 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.675340 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.675345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.675380 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:27.675515 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.675524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.677012 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.677017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.678388 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.678416 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.679432 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.679436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.679541 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.679578 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.679698 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.679809 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":67,"spinlock_wait_cycles":3584,"thread_start_us":91,"threads_started":1}
I20260620 06:15:27.679962 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.674269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.14.209.129:36829)
0620 06:15:27.674851 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.674862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.674884 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:15:27.674958 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.674963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.675169 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.675681 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:27.675846 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.676906 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.678659 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.679302 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.679433 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.679555 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.679741 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.679839 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":119,"threads_started":1}
I20260620 06:15:27.680485 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.673941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:32953 (local address 127.0.0.1:33710)
0620 06:15:27.674902 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.674914 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.675043 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.675709 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.675711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.675725 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:27.675788 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.675791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.676843 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.676847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.678399 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.678411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.680082 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.680085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.680178 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.680200 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.680279 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.680348 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":105,"thread_start_us":95,"threads_started":1}
I20260620 06:15:27.680858 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.673632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.14.209.131:35543)
0620 06:15:27.674726 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.674737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.674757 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:15:27.674830 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.674836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.675031 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.675193 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:27.675640 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.676825 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.679198 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.680211 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.680327 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.680433 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.680606 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.680706 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":118,"threads_started":1}
I20260620 06:15:27.680747 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.673535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:35543 (local address 127.0.0.1:40774)
0620 06:15:27.674290 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.674334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.674467 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.675338 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.675344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.675376 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:27.675505 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.675511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.676948 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.676951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.678399 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.678411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.680352 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.680355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.680460 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.680482 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.680561 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.680623 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":101,"spinlock_wait_cycles":9472,"thread_start_us":72,"threads_started":1}
I20260620 06:15:27.681102 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.674325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.14.209.130:32953)
0620 06:15:27.674775 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.674787 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.674981 (+ 194us) server_negotiation.cc:407] Connection header received
0620 06:15:27.675211 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.675216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.675383 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.675525 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:27.675899 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.676717 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.678601 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.679755 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.679887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.679999 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.680843 (+ 844us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.680958 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":113,"threads_started":1}
I20260620 06:15:27.682613 950 tablet_service.cc:1467] Processing CreateTablet for tablet 1283479cf32d43999b50425ac4087841 (DEFAULT_TABLE table=test-table [id=bf746930437f4a82881acf0652adf538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:27.683498 950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1283479cf32d43999b50425ac4087841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:27.683306 1218 tablet_service.cc:1467] Processing CreateTablet for tablet 1283479cf32d43999b50425ac4087841 (DEFAULT_TABLE table=test-table [id=bf746930437f4a82881acf0652adf538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:27.684268 1218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1283479cf32d43999b50425ac4087841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:27.685639 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 1283479cf32d43999b50425ac4087841 (DEFAULT_TABLE table=test-table [id=bf746930437f4a82881acf0652adf538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:27.686533 1084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1283479cf32d43999b50425ac4087841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:27.689565 1301 tablet_bootstrap.cc:492] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: Bootstrap starting.
I20260620 06:15:27.692366 1301 tablet_bootstrap.cc:654] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:27.693017 1303 tablet_bootstrap.cc:492] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Bootstrap starting.
I20260620 06:15:27.693552 1302 tablet_bootstrap.cc:492] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Bootstrap starting.
I20260620 06:15:27.693604 1301 log.cc:824] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:27.695120 1303 tablet_bootstrap.cc:654] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:27.695291 1301 tablet_bootstrap.cc:492] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: No bootstrap required, opened a new log
I20260620 06:15:27.695484 1301 ts_tablet_manager.cc:1397] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 06:15:27.695562 1302 tablet_bootstrap.cc:654] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:27.696406 1303 log.cc:824] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:27.696772 1302 log.cc:824] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:27.698208 1303 tablet_bootstrap.cc:492] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: No bootstrap required, opened a new log
I20260620 06:15:27.698227 1302 tablet_bootstrap.cc:492] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: No bootstrap required, opened a new log
I20260620 06:15:27.698381 1303 ts_tablet_manager.cc:1397] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 06:15:27.698406 1302 ts_tablet_manager.cc:1397] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 06:15:27.699342 1301 raft_consensus.cc:348] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.699596 1301 raft_consensus.cc:374] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:27.699641 1301 raft_consensus.cc:729] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3008595d95c4772bfb83eeeb0647740, State: Initialized, Role: FOLLOWER
I20260620 06:15:27.700057 1301 consensus_queue.cc:260] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.700727 1016 heartbeater.cc:503] Master 127.14.209.190:38941 was elected leader, sending a full tablet report...
I20260620 06:15:27.701006 1301 ts_tablet_manager.cc:1428] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:15:27.701946 1303 raft_consensus.cc:348] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.701990 1302 raft_consensus.cc:348] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.702167 1302 raft_consensus.cc:374] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:27.702172 1303 raft_consensus.cc:374] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:27.702198 1302 raft_consensus.cc:729] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad5bd227192448e846c32e531fe9a5c, State: Initialized, Role: FOLLOWER
I20260620 06:15:27.702217 1303 raft_consensus.cc:729] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c94f47578d4d599e0c25f71dfdcee5, State: Initialized, Role: FOLLOWER
I20260620 06:15:27.702549 1302 consensus_queue.cc:260] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.702637 1303 consensus_queue.cc:260] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.703461 1302 ts_tablet_manager.cc:1428] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 06:15:27.703640 1284 heartbeater.cc:503] Master 127.14.209.190:38941 was elected leader, sending a full tablet report...
I20260620 06:15:27.704046 1150 heartbeater.cc:503] Master 127.14.209.190:38941 was elected leader, sending a full tablet report...
I20260620 06:15:27.704279 1303 ts_tablet_manager.cc:1428] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260620 06:15:27.717839 1017 tablet.cc:2369] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:27.882478 1285 tablet.cc:2369] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:27.924376 1309 raft_consensus.cc:482] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:27.924557 1309 raft_consensus.cc:504] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.925647 1309 leader_election.cc:288] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ad5bd227192448e846c32e531fe9a5c (127.14.209.131:35543), f3008595d95c4772bfb83eeeb0647740 (127.14.209.129:36829)
W20260620 06:15:27.928411 1151 tablet.cc:2369] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:27.929349 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.925840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:35543 (local address 127.14.209.130:57915)
0620 06:15:27.925998 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.926013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.926101 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.926403 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.926406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.926421 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:27.926557 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.926563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.927332 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.927335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.928168 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.928177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.928953 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.928956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.929065 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.929085 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.929158 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.929221 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 06:15:27.929396 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.925962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:57915 (local address 127.14.209.131:35543)
0620 06:15:27.926154 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.926161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.926179 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:15:27.926238 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.926241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.926292 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.926390 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:27.926672 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.927234 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.928299 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.928855 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.928969 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.929024 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.929187 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.929257 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 06:15:27.929541 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.926018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:43259 (local address 127.14.209.129:36829)
0620 06:15:27.926135 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.926142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.926255 (+ 113us) server_negotiation.cc:407] Connection header received
0620 06:15:27.926383 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.926387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.926438 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.926516 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:27.926788 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.927335 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.928404 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.928947 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.929056 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.929104 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.929337 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.929415 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:15:27.929865 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.925885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:36829 (local address 127.14.209.130:43259)
0620 06:15:27.926152 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:27.926165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:27.926272 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:27.926538 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:27.926540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:27.926555 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:27.926678 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.926684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.927438 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.927441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.928212 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:27.928219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.929089 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.929092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:27.929210 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.929231 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:15:27.929307 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:15:27.929369 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":67,"threads_started":1}
I20260620 06:15:27.929946 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283479cf32d43999b50425ac4087841" candidate_uuid: "87c94f47578d4d599e0c25f71dfdcee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ad5bd227192448e846c32e531fe9a5c" is_pre_election: true
I20260620 06:15:27.930238 1238 raft_consensus.cc:2454] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87c94f47578d4d599e0c25f71dfdcee5 in term 0.
I20260620 06:15:27.930402 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283479cf32d43999b50425ac4087841" candidate_uuid: "87c94f47578d4d599e0c25f71dfdcee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3008595d95c4772bfb83eeeb0647740" is_pre_election: true
I20260620 06:15:27.930671 970 raft_consensus.cc:2454] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87c94f47578d4d599e0c25f71dfdcee5 in term 0.
I20260620 06:15:27.930719 1039 leader_election.cc:302] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ad5bd227192448e846c32e531fe9a5c, 87c94f47578d4d599e0c25f71dfdcee5; no voters:
I20260620 06:15:27.931013 1309 raft_consensus.cc:2781] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:15:27.931124 1309 raft_consensus.cc:482] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:15:27.931170 1309 raft_consensus.cc:3037] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:27.932155 1309 raft_consensus.cc:504] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.932487 1309 leader_election.cc:288] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [CANDIDATE]: Term 1 election: Requested vote from peers 1ad5bd227192448e846c32e531fe9a5c (127.14.209.131:35543), f3008595d95c4772bfb83eeeb0647740 (127.14.209.129:36829)
I20260620 06:15:27.932813 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283479cf32d43999b50425ac4087841" candidate_uuid: "87c94f47578d4d599e0c25f71dfdcee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ad5bd227192448e846c32e531fe9a5c"
I20260620 06:15:27.932926 1238 raft_consensus.cc:3037] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:27.934144 1238 raft_consensus.cc:2454] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c94f47578d4d599e0c25f71dfdcee5 in term 1.
I20260620 06:15:27.934407 1039 leader_election.cc:302] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ad5bd227192448e846c32e531fe9a5c, 87c94f47578d4d599e0c25f71dfdcee5; no voters:
I20260620 06:15:27.934564 1309 raft_consensus.cc:2781] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:27.934759 1309 raft_consensus.cc:686] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 1 LEADER]: Becoming Leader. State: Replica: 87c94f47578d4d599e0c25f71dfdcee5, State: Running, Role: LEADER
I20260620 06:15:27.935093 1309 consensus_queue.cc:237] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } }
I20260620 06:15:27.938236 824 catalog_manager.cc:5495] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87c94f47578d4d599e0c25f71dfdcee5 (127.14.209.130). New cstate: current_term: 1 leader_uuid: "87c94f47578d4d599e0c25f71dfdcee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87c94f47578d4d599e0c25f71dfdcee5" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 32953 } health_report { overall_health: HEALTHY } } }
I20260620 06:15:27.939458 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283479cf32d43999b50425ac4087841" candidate_uuid: "87c94f47578d4d599e0c25f71dfdcee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3008595d95c4772bfb83eeeb0647740"
I20260620 06:15:27.939597 970 raft_consensus.cc:3037] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:27.940761 970 raft_consensus.cc:2454] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c94f47578d4d599e0c25f71dfdcee5 in term 1.
I20260620 06:15:27.967837 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:27.964437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.14.209.130:32953)
0620 06:15:27.964569 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:27.964573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:27.964585 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:15:27.964730 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:27.964735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:27.964796 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:27.964898 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:15:27.965122 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.965604 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.966426 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:27.966641 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:27.966743 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:27.967088 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:15:27.967428 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:15:27.967573 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:27.967668 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 06:15:27.967699 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:15:27.972855 970 raft_consensus.cc:1261] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 1 FOLLOWER]: Refusing update from remote peer 87c94f47578d4d599e0c25f71dfdcee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:27.972859 1238 raft_consensus.cc:1261] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 1 FOLLOWER]: Refusing update from remote peer 87c94f47578d4d599e0c25f71dfdcee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:27.973457 1311 consensus_queue.cc:1035] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3008595d95c4772bfb83eeeb0647740" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:27.973605 1309 consensus_queue.cc:1035] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad5bd227192448e846c32e531fe9a5c" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 35543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:27.979835 1314 mvcc.cc:205] Tried to move back new op lower bound from 7298810380171100160 to 7298810380024668160. Current Snapshot: MvccSnapshot[applied={T|T < 7298810380171100160}]
I20260620 06:15:27.980413 1315 mvcc.cc:205] Tried to move back new op lower bound from 7298810380171100160 to 7298810380024668160. Current Snapshot: MvccSnapshot[applied={T|T < 7298810380171100160}]
I20260620 06:15:27.989074 824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:27.989228 824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:27.991289 824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2defda74b6174d81a6d59a0ba24ac461: Sending DeleteTablet for 3 replicas of tablet 1283479cf32d43999b50425ac4087841
I20260620 06:15:27.991992 1218 tablet_service.cc:1514] Processing DeleteTablet for tablet 1283479cf32d43999b50425ac4087841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:27 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:40774
I20260620 06:15:27.992141 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 1283479cf32d43999b50425ac4087841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:27 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:33710
I20260620 06:15:27.992470 950 tablet_service.cc:1514] Processing DeleteTablet for tablet 1283479cf32d43999b50425ac4087841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:27 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:40040
I20260620 06:15:27.992681 1324 tablet_replica.cc:330] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: stopping tablet replica
I20260620 06:15:27.992789 1323 tablet_replica.cc:330] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: stopping tablet replica
I20260620 06:15:27.993059 1325 tablet_replica.cc:330] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740: stopping tablet replica
I20260620 06:15:27.993067 1323 raft_consensus.cc:2229] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:27.993085 1324 raft_consensus.cc:2229] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:15:27.993309 1325 raft_consensus.cc:2229] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:27.993343 1323 raft_consensus.cc:2258] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:27.993410 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 887
I20260620 06:15:27.993561 1325 raft_consensus.cc:2258] T 1283479cf32d43999b50425ac4087841 P f3008595d95c4772bfb83eeeb0647740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:27.993785 1324 raft_consensus.cc:2258] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:27.995138 1324 ts_tablet_manager.cc:1905] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:27.995138 1323 ts_tablet_manager.cc:1905] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:27.997695 1324 ts_tablet_manager.cc:1918] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:27.997695 1323 ts_tablet_manager.cc:1918] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:27.997787 1323 log.cc:1196] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/1283479cf32d43999b50425ac4087841
I20260620 06:15:27.997905 1324 log.cc:1196] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/1283479cf32d43999b50425ac4087841
I20260620 06:15:27.998148 1323 ts_tablet_manager.cc:1939] T 1283479cf32d43999b50425ac4087841 P 1ad5bd227192448e846c32e531fe9a5c: Deleting consensus metadata
I20260620 06:15:27.998306 1324 ts_tablet_manager.cc:1939] T 1283479cf32d43999b50425ac4087841 P 87c94f47578d4d599e0c25f71dfdcee5: Deleting consensus metadata
I20260620 06:15:27.999320 809 catalog_manager.cc:4841] TS 87c94f47578d4d599e0c25f71dfdcee5 (127.14.209.130:32953): tablet 1283479cf32d43999b50425ac4087841 (table test-table [id=bf746930437f4a82881acf0652adf538]) successfully deleted
I20260620 06:15:27.999308 811 catalog_manager.cc:4841] TS 1ad5bd227192448e846c32e531fe9a5c (127.14.209.131:35543): tablet 1283479cf32d43999b50425ac4087841 (table test-table [id=bf746930437f4a82881acf0652adf538]) successfully deleted
W20260620 06:15:28.001601 812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:36829 (error 108)
I20260620 06:15:28.001816 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1021
I20260620 06:15:28.003178 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.002905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:36829 (local address 127.0.0.1:40044)
0620 06:15:28.003021 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.003104 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:36829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:15:28.003373 812 catalog_manager.cc:4584] TS f3008595d95c4772bfb83eeeb0647740 (127.14.209.129:36829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1283479cf32d43999b50425ac4087841: Network error: Client connection negotiation failed: client connection to 127.14.209.129:36829: connect: Connection refused (error 111)
I20260620 06:15:28.006945 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1155
I20260620 06:15:28.012398 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 793
2026-06-20T06:15:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2400 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T06:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:28Z Disabled control of system clock
I20260620 06:15:28.041443 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46489
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:15:28.146652 1330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:28.146880 1330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:28.146914 1330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:28.150941 1330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:28.150995 1330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:28.151011 1330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:28.151028 1330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:28.151099 1330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:28.155961 1330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:46489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:28.156857 1330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:28.158082 1330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:15:28.166271 1330 server_base.cc:1034] running on GCE node
W20260620 06:15:28.166302 1335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:28.166302 1336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:28.166468 1338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:28.166991 1330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:28.168114 1330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:28.169322 1330 hybrid_clock.cc:648] HybridClock initialized: now 1781936128169300 us; error 42 us; skew 500 ppm
I20260620 06:15:28.179879 1330 webserver.cc:460] Webserver started at http://127.14.209.190:35787/ using document root <none> and password file <none>
I20260620 06:15:28.180466 1330 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:28.180555 1330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:28.180773 1330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:28.182729 1330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "759f861a952e47408580e0a00a06cce5"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "e27f7fd7ab2d16b6e5e37cd8e76f063a"
server_key_iv: "9f870c6a7a5eda043b6405c5d0cd589c"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.183290 1330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "759f861a952e47408580e0a00a06cce5"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "e27f7fd7ab2d16b6e5e37cd8e76f063a"
server_key_iv: "9f870c6a7a5eda043b6405c5d0cd589c"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.186692 1330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:15:28.189008 1344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.190178 1330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:15:28.190323 1330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "759f861a952e47408580e0a00a06cce5"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "e27f7fd7ab2d16b6e5e37cd8e76f063a"
server_key_iv: "9f870c6a7a5eda043b6405c5d0cd589c"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.190425 1330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:28.205896 1330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:28.206529 1330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:28.206707 1330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:28.213150 1330 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:46489
I20260620 06:15:28.213164 1396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:46489 every 8 connection(s)
I20260620 06:15:28.214026 1330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:28.216696 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1330
I20260620 06:15:28.216840 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:28.216863 1397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:28.217172 15174 external_mini_cluster.cc:1428] Setting key c85555fd81073c9ccfc956f2cd452c10
I20260620 06:15:28.223371 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5: Bootstrap starting.
I20260620 06:15:28.225404 1397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:28.225669 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.218547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.14.209.190:46489)
0620 06:15:28.219137 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.219160 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.219194 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:15:28.219829 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.219845 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.220187 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.220489 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:28.220882 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.221802 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.222656 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.223164 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.223325 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.224124 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:28.224129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:28.224140 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:28.224481 (+ 341us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:28.224531 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:28.224550 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:28.224661 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.224770 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.224997 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260620 06:15:28.226141 1397 log.cc:824] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:28.227931 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5: No bootstrap required, opened a new log
I20260620 06:15:28.231258 1397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759f861a952e47408580e0a00a06cce5" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46489 } }
I20260620 06:15:28.231472 1397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:28.231541 1397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759f861a952e47408580e0a00a06cce5, State: Initialized, Role: FOLLOWER
I20260620 06:15:28.231918 1397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759f861a952e47408580e0a00a06cce5" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46489 } }
I20260620 06:15:28.232055 1397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:28.232131 1397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:28.232242 1397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:28.233007 1397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759f861a952e47408580e0a00a06cce5" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46489 } }
I20260620 06:15:28.233317 1397 leader_election.cc:302] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759f861a952e47408580e0a00a06cce5; no voters:
I20260620 06:15:28.233592 1397 leader_election.cc:288] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:28.233722 1402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:28.233968 1402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [term 1 LEADER]: Becoming Leader. State: Replica: 759f861a952e47408580e0a00a06cce5, State: Running, Role: LEADER
I20260620 06:15:28.234220 1402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759f861a952e47408580e0a00a06cce5" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46489 } }
I20260620 06:15:28.234678 1397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:28.235790 1404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759f861a952e47408580e0a00a06cce5. Latest consensus state: current_term: 1 leader_uuid: "759f861a952e47408580e0a00a06cce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759f861a952e47408580e0a00a06cce5" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46489 } } }
I20260620 06:15:28.235987 1404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:28.236228 1403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759f861a952e47408580e0a00a06cce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759f861a952e47408580e0a00a06cce5" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 46489 } } }
I20260620 06:15:28.236325 1403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:28.236860 1411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:28.239257 1411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:28.244055 1411 catalog_manager.cc:1347] Generated new cluster ID: 2e8db713e5ee4b3188763488d0ea4723
I20260620 06:15:28.244112 1411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:28.253476 1411 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:28.254571 1411 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:28.261600 1411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5: Generated new TSK 0
I20260620 06:15:28.262287 1411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:28.272711 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46489
--builtin_ntp_servers=127.14.209.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:15:28.378851 1421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:28.379114 1421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:28.379186 1421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:28.383128 1421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:28.383194 1421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:28.383283 1421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:28.387850 1421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:28.388751 1421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:28.389981 1421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:28.398850 1429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:28.398857 1426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:28.399032 1421 server_base.cc:1034] running on GCE node
W20260620 06:15:28.399246 1427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:28.399637 1421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:28.400195 1421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:28.401351 1421 hybrid_clock.cc:648] HybridClock initialized: now 1781936128401324 us; error 41 us; skew 500 ppm
I20260620 06:15:28.402962 1421 webserver.cc:460] Webserver started at http://127.14.209.129:40187/ using document root <none> and password file <none>
I20260620 06:15:28.403510 1421 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:28.403569 1421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:28.403782 1421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:28.405647 1421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "32390c92c5f24a8fa2045fe39137363b"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "1f51ab07d26050ee6e68c105c509f70b"
server_key_iv: "2e78fa56850130a8ed55eba17b905185"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.406167 1421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "32390c92c5f24a8fa2045fe39137363b"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "1f51ab07d26050ee6e68c105c509f70b"
server_key_iv: "2e78fa56850130a8ed55eba17b905185"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.409513 1421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 06:15:28.411703 1435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.412776 1421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:15:28.412901 1421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "32390c92c5f24a8fa2045fe39137363b"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "1f51ab07d26050ee6e68c105c509f70b"
server_key_iv: "2e78fa56850130a8ed55eba17b905185"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.413000 1421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:28.426626 1421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:28.427234 1421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:28.427412 1421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:28.427915 1421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:28.428743 1421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:28.428815 1421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.428879 1421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:28.428925 1421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.437485 1421 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:41227
I20260620 06:15:28.437516 1548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:41227 every 8 connection(s)
I20260620 06:15:28.438246 1421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:28.438426 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1421
I20260620 06:15:28.438551 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:15:28.438920 15174 external_mini_cluster.cc:1428] Setting key 357b812df84a7ac44442eb2fef23dd21
I20260620 06:15:28.440903 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46489
--builtin_ntp_servers=127.14.209.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:15:28.446978 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.439867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:36699 (local address 127.14.209.190:46489)
0620 06:15:28.439988 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.439991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.441119 (+ 1128us) server_negotiation.cc:407] Connection header received
0620 06:15:28.442437 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.442442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.442505 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.442603 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:28.443457 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.443989 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.445121 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.445370 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.445479 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.446500 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:28.446504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:28.446507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:28.446519 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:28.446543 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:28.446551 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:28.446620 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.446776 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.446841 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:15:28.447700 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.440342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46489 (local address 127.14.209.129:36699)
0620 06:15:28.440984 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.441031 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:28.442247 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:28.442736 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:28.442741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:28.442831 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:28.443323 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.443336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.444114 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.444117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.444995 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.445004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.445513 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.445517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.445681 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.446251 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:28.446253 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:28.446289 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:28.446344 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:28.446635 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:28.446643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:28.446653 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 06:15:28.446762 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 06:15:28.447009 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":132,"threads_started":1}
I20260620 06:15:28.448956 1549 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46489
I20260620 06:15:28.449167 1549 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:28.449635 1549 heartbeater.cc:511] Master 127.14.209.190:46489 requested a full tablet report, sending...
I20260620 06:15:28.451020 1361 ts_manager.cc:194] Registered new tserver with Master: 32390c92c5f24a8fa2045fe39137363b (127.14.209.129:41227)
I20260620 06:15:28.452147 1361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.129:36699
W20260620 06:15:28.547005 1552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:28.547257 1552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:28.547322 1552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:28.551164 1552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:28.551226 1552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:28.551306 1552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:28.555933 1552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:28.556802 1552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:28.557961 1552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:28.566588 1558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:28.566632 1552 server_base.cc:1034] running on GCE node
W20260620 06:15:28.566653 1557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:28.566615 1560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:28.567260 1552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:28.567831 1552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:28.568997 1552 hybrid_clock.cc:648] HybridClock initialized: now 1781936128568985 us; error 31 us; skew 500 ppm
I20260620 06:15:28.570612 1552 webserver.cc:460] Webserver started at http://127.14.209.130:34385/ using document root <none> and password file <none>
I20260620 06:15:28.571168 1552 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:28.571257 1552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:28.571462 1552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:28.573268 1552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "475920a79b834257a577f7dec853724a"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "5f6b0fef6d8c3e278fbeb7b2f5a0ce7e"
server_key_iv: "9f8a4bb19cfa87dd70033fac9746785d"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.573770 1552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "475920a79b834257a577f7dec853724a"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "5f6b0fef6d8c3e278fbeb7b2f5a0ce7e"
server_key_iv: "9f8a4bb19cfa87dd70033fac9746785d"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.576993 1552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 06:15:28.579345 1566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.580442 1552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:15:28.580578 1552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "475920a79b834257a577f7dec853724a"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "5f6b0fef6d8c3e278fbeb7b2f5a0ce7e"
server_key_iv: "9f8a4bb19cfa87dd70033fac9746785d"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.580679 1552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:28.597833 1552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:28.598367 1552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:28.598531 1552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:28.599012 1552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:28.599845 1552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:28.599910 1552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.599972 1552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:28.600019 1552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.608500 1552 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:46401
I20260620 06:15:28.608528 1679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:46401 every 8 connection(s)
I20260620 06:15:28.609236 1552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:28.616545 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1552
I20260620 06:15:28.616674 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:15:28.617065 15174 external_mini_cluster.cc:1428] Setting key 754125c547a6140da5949d98df8ae454
I20260620 06:15:28.617533 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.610825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:54289 (local address 127.14.209.190:46489)
0620 06:15:28.610960 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.610964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.611902 (+ 938us) server_negotiation.cc:407] Connection header received
0620 06:15:28.613152 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.613154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.613198 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.613269 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:28.614055 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.614558 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.615632 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.615848 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.615926 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.617046 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:28.617050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:28.617052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:28.617068 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:28.617085 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:28.617093 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:28.617174 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.617342 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.617398 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:15:28.618322 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.611170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46489 (local address 127.14.209.130:54289)
0620 06:15:28.611777 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.611813 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:28.612983 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:28.613403 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:28.613409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:28.613480 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:28.613929 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.613940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.614681 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.614684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.615524 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.615533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.615974 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.615977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.616116 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.616788 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:28.616789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:28.616847 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:28.616902 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:28.617195 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:28.617203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:28.617214 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 06:15:28.617335 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 06:15:28.617573 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260620 06:15:28.618813 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46489
--builtin_ntp_servers=127.14.209.148:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:15:28.619477 1680 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46489
I20260620 06:15:28.619683 1680 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:28.620113 1680 heartbeater.cc:511] Master 127.14.209.190:46489 requested a full tablet report, sending...
I20260620 06:15:28.621052 1361 ts_manager.cc:194] Registered new tserver with Master: 475920a79b834257a577f7dec853724a (127.14.209.130:46401)
I20260620 06:15:28.621588 1361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.130:54289
W20260620 06:15:28.725620 1683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:28.725836 1683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:28.725872 1683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:28.729875 1683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:28.729920 1683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:28.729997 1683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:28.734571 1683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:46489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:28.735502 1683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:28.736871 1683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:28.745831 1688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:28.745872 1691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:28.745872 1689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:28.745970 1683 server_base.cc:1034] running on GCE node
I20260620 06:15:28.746381 1683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:28.746935 1683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:28.748108 1683 hybrid_clock.cc:648] HybridClock initialized: now 1781936128748089 us; error 26 us; skew 500 ppm
I20260620 06:15:28.749725 1683 webserver.cc:460] Webserver started at http://127.14.209.131:37601/ using document root <none> and password file <none>
I20260620 06:15:28.750257 1683 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:28.750315 1683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:28.750523 1683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:28.752424 1683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "593e207e786d4d4899344b464b2c9483"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "b588984e4b65719780841b195377c5a0"
server_key_iv: "b13ccc6c14eadb2630c6f567cb765dc4"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.752923 1683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "593e207e786d4d4899344b464b2c9483"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "b588984e4b65719780841b195377c5a0"
server_key_iv: "b13ccc6c14eadb2630c6f567cb765dc4"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.756680 1683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 06:15:28.758838 1697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.759949 1683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:28.760066 1683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "593e207e786d4d4899344b464b2c9483"
format_stamp: "Formatted at 2026-06-20 06:15:28 on dist-test-slave-zt01"
server_key: "b588984e4b65719780841b195377c5a0"
server_key_iv: "b13ccc6c14eadb2630c6f567cb765dc4"
server_key_version: "encryptionkey@0"
I20260620 06:15:28.760169 1683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:28.771605 1683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:28.772135 1683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:28.772301 1683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:28.772797 1683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:28.773597 1683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:28.773662 1683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.773731 1683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:28.773777 1683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:28.782145 1683 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:42231
I20260620 06:15:28.782178 1810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:42231 every 8 connection(s)
I20260620 06:15:28.782917 1683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:28.784819 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1683
I20260620 06:15:28.784932 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:15:28.785214 15174 external_mini_cluster.cc:1428] Setting key 9fa2b264614f5bbdaaae3133795def8a
I20260620 06:15:28.791486 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.784625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:52797 (local address 127.14.209.190:46489)
0620 06:15:28.784757 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.784762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.785832 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 06:15:28.787475 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.787478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.787521 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.787585 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:28.788324 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.788837 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.789884 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.790086 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.790159 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.791080 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:28.791085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:28.791092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:28.791117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:28.791141 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:28.791149 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:28.791206 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.791338 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.791383 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:15:28.792204 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.784964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:46489 (local address 127.14.209.131:52797)
0620 06:15:28.785667 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.785716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:28.787317 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:28.787695 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:28.787700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:28.787771 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:28.788210 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.788222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.788943 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.788946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.789780 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.789789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.790213 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.790216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.790347 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.790832 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:28.790833 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:28.790867 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:28.790920 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:28.791222 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:28.791228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:28.791232 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:15:28.791324 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 06:15:28.791507 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260620 06:15:28.793249 1811 heartbeater.cc:347] Connected to a master server at 127.14.209.190:46489
I20260620 06:15:28.793431 1811 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:28.793812 1811 heartbeater.cc:511] Master 127.14.209.190:46489 requested a full tablet report, sending...
I20260620 06:15:28.794672 1361 ts_manager.cc:194] Registered new tserver with Master: 593e207e786d4d4899344b464b2c9483 (127.14.209.131:42231)
I20260620 06:15:28.795243 1361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.131:52797
I20260620 06:15:28.798617 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:15:28.803892 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.800317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.14.209.190:46489)
0620 06:15:28.800434 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.800439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.800621 (+ 182us) server_negotiation.cc:407] Connection header received
0620 06:15:28.800690 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.800692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.800729 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.800791 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:28.801020 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.801483 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.802265 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.802471 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.802556 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.803308 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:28.803311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:28.803313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:28.803338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:28.803360 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:28.803374 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:28.803466 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.803707 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.803742 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:15:28.806397 1361 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:15:28.809258 1361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:15:28.823031 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.816567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:42231 (local address 127.0.0.1:36390)
0620 06:15:28.817331 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.817376 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:28.817507 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:28.818326 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:28.818334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:28.818358 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:28.818493 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.818500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.820162 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.820164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.821511 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.821520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.822544 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.822547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.822703 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.822749 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 06:15:28.822847 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:15:28.822925 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":179,"spinlock_wait_cycles":12544,"thread_start_us":75,"threads_started":1}
I20260620 06:15:28.823148 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.816632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.14.209.131:42231)
0620 06:15:28.817854 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.817863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.817902 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:28.817969 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.817974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.818145 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.818637 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:28.818823 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.819991 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.821628 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.822410 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.822550 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.822713 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.822893 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.822978 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":144,"threads_started":1}
I20260620 06:15:28.823462 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.816670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41227 (local address 127.0.0.1:59950)
0620 06:15:28.817330 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.817377 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:28.817505 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:28.818676 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:28.818679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:28.818696 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:28.818775 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.818780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.820181 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.820184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.821622 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.821632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.822558 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.822561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.823185 (+ 624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.823212 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:15:28.823312 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:15:28.823370 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":594,"spinlock_wait_cycles":2944,"thread_start_us":87,"threads_started":1}
I20260620 06:15:28.823769 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.817077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46401 (local address 127.0.0.1:54940)
0620 06:15:28.817487 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:28.817500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:28.817572 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:28.818506 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:28.818508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:28.818525 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:28.818614 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.818619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.820126 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.820131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.821809 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:28.821815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.823384 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.823388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:28.823481 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.823506 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:15:28.823579 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:15:28.823640 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":191,"thread_start_us":88,"threads_started":1}
I20260620 06:15:28.823992 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.817080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.14.209.129:41227)
0620 06:15:28.817976 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.817987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.818006 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:15:28.818358 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.818366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.818532 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.818679 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:28.818869 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.820056 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.821731 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.822432 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.822698 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.822860 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.823723 (+ 863us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.823828 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":119,"threads_started":1}
I20260620 06:15:28.824591 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:28.817492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.14.209.130:46401)
0620 06:15:28.817909 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:28.817920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:28.817943 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:15:28.818020 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:28.818025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:28.818207 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:28.818439 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:28.818767 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.819991 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.821910 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:28.822421 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:28.822563 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:28.822721 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:28.823916 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0620 06:15:28.824013 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":123,"threads_started":1}
I20260620 06:15:28.825917 1744 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9b0c863a344f698ae3c7039f5f1af8 (DEFAULT_TABLE table=test-table [id=c541256979d541fc912e664f40d12c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:28.826007 1483 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9b0c863a344f698ae3c7039f5f1af8 (DEFAULT_TABLE table=test-table [id=c541256979d541fc912e664f40d12c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:28.826717 1483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d9b0c863a344f698ae3c7039f5f1af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:28.826717 1744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d9b0c863a344f698ae3c7039f5f1af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:28.829505 1614 tablet_service.cc:1467] Processing CreateTablet for tablet 6d9b0c863a344f698ae3c7039f5f1af8 (DEFAULT_TABLE table=test-table [id=c541256979d541fc912e664f40d12c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:28.830417 1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d9b0c863a344f698ae3c7039f5f1af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:28.832402 1828 tablet_bootstrap.cc:492] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Bootstrap starting.
I20260620 06:15:28.834062 1829 tablet_bootstrap.cc:492] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Bootstrap starting.
I20260620 06:15:28.834472 1828 tablet_bootstrap.cc:654] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:28.835395 1828 log.cc:824] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:28.836180 1829 tablet_bootstrap.cc:654] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:28.836932 1829 log.cc:824] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:28.836938 1831 tablet_bootstrap.cc:492] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Bootstrap starting.
I20260620 06:15:28.837805 1828 tablet_bootstrap.cc:492] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: No bootstrap required, opened a new log
I20260620 06:15:28.838009 1828 ts_tablet_manager.cc:1397] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:15:28.838572 1829 tablet_bootstrap.cc:492] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: No bootstrap required, opened a new log
I20260620 06:15:28.838747 1829 ts_tablet_manager.cc:1397] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:15:28.838990 1831 tablet_bootstrap.cc:654] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:28.839761 1831 log.cc:824] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:28.841619 1828 raft_consensus.cc:348] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:28.841804 1828 raft_consensus.cc:374] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:28.841883 1828 raft_consensus.cc:729] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32390c92c5f24a8fa2045fe39137363b, State: Initialized, Role: FOLLOWER
I20260620 06:15:28.842051 1829 raft_consensus.cc:348] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:28.842247 1829 raft_consensus.cc:374] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:28.842299 1829 raft_consensus.cc:729] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593e207e786d4d4899344b464b2c9483, State: Initialized, Role: FOLLOWER
I20260620 06:15:28.842309 1828 consensus_queue.cc:260] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:28.842685 1829 consensus_queue.cc:260] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:28.842965 1549 heartbeater.cc:503] Master 127.14.209.190:46489 was elected leader, sending a full tablet report...
I20260620 06:15:28.843248 1828 ts_tablet_manager.cc:1428] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 06:15:28.843433 1811 heartbeater.cc:503] Master 127.14.209.190:46489 was elected leader, sending a full tablet report...
I20260620 06:15:28.843585 1829 ts_tablet_manager.cc:1428] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:15:28.845815 1831 tablet_bootstrap.cc:492] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: No bootstrap required, opened a new log
I20260620 06:15:28.845996 1831 ts_tablet_manager.cc:1397] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 06:15:28.849301 1831 raft_consensus.cc:348] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:28.849503 1831 raft_consensus.cc:374] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:28.849560 1831 raft_consensus.cc:729] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475920a79b834257a577f7dec853724a, State: Initialized, Role: FOLLOWER
I20260620 06:15:28.849895 1831 consensus_queue.cc:260] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:28.850425 1680 heartbeater.cc:503] Master 127.14.209.190:46489 was elected leader, sending a full tablet report...
I20260620 06:15:28.850725 1831 ts_tablet_manager.cc:1428] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 06:15:28.860078 1681 tablet.cc:2369] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:28.939347 1550 tablet.cc:2369] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:29.003744 1834 raft_consensus.cc:482] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:29.003981 1834 raft_consensus.cc:504] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:29.005089 1834 leader_election.cc:288] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593e207e786d4d4899344b464b2c9483 (127.14.209.131:42231), 475920a79b834257a577f7dec853724a (127.14.209.130:46401)
I20260620 06:15:29.008759 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.005425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:53683 (local address 127.14.209.130:46401)
0620 06:15:29.005552 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:29.005556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:29.005715 (+ 159us) server_negotiation.cc:407] Connection header received
0620 06:15:29.005858 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:29.005861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:29.005911 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:29.005990 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:29.006256 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.006743 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.007840 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.008230 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.008353 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:29.008402 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:29.008564 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 06:15:29.008629 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:15:29.008801 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.005298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46401 (local address 127.14.209.129:53683)
0620 06:15:29.005629 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:29.005640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:29.005755 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:29.006000 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:29.006003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:29.006018 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:29.006149 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:29.006154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.006844 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.006850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:29.007723 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:29.007732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.008344 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.008347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:29.008441 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:29.008468 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:15:29.008550 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:15:29.008618 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":77,"threads_started":1}
I20260620 06:15:29.008795 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.005402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:35119 (local address 127.14.209.131:42231)
0620 06:15:29.005594 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:29.005600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:29.005685 (+ 85us) server_negotiation.cc:407] Connection header received
0620 06:15:29.005843 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:29.005852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:29.005895 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:29.005970 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:29.006247 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.006756 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.007855 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.008242 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.008386 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:29.008432 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:29.008605 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 06:15:29.008668 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 06:15:29.009018 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.005273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:42231 (local address 127.14.209.129:35119)
0620 06:15:29.005590 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:29.005604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:29.005737 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:29.005991 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:29.005994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:29.006009 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:29.006139 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:29.006144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.006878 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.006884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:29.007726 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:29.007735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.008387 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.008390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:29.008486 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:29.008508 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:15:29.008590 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:15:29.008657 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":10,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260620 06:15:29.009397 1634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9b0c863a344f698ae3c7039f5f1af8" candidate_uuid: "32390c92c5f24a8fa2045fe39137363b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475920a79b834257a577f7dec853724a" is_pre_election: true
I20260620 06:15:29.009584 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9b0c863a344f698ae3c7039f5f1af8" candidate_uuid: "32390c92c5f24a8fa2045fe39137363b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593e207e786d4d4899344b464b2c9483" is_pre_election: true
I20260620 06:15:29.009666 1634 raft_consensus.cc:2454] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32390c92c5f24a8fa2045fe39137363b in term 0.
I20260620 06:15:29.009856 1765 raft_consensus.cc:2454] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32390c92c5f24a8fa2045fe39137363b in term 0.
I20260620 06:15:29.010152 1439 leader_election.cc:302] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32390c92c5f24a8fa2045fe39137363b, 475920a79b834257a577f7dec853724a; no voters:
I20260620 06:15:29.010416 1834 raft_consensus.cc:2781] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:15:29.010494 1834 raft_consensus.cc:482] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:15:29.010541 1834 raft_consensus.cc:3037] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:29.011586 1834 raft_consensus.cc:504] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:29.011940 1834 leader_election.cc:288] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [CANDIDATE]: Term 1 election: Requested vote from peers 593e207e786d4d4899344b464b2c9483 (127.14.209.131:42231), 475920a79b834257a577f7dec853724a (127.14.209.130:46401)
I20260620 06:15:29.012310 1765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9b0c863a344f698ae3c7039f5f1af8" candidate_uuid: "32390c92c5f24a8fa2045fe39137363b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593e207e786d4d4899344b464b2c9483"
I20260620 06:15:29.012432 1765 raft_consensus.cc:3037] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:29.012444 1634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d9b0c863a344f698ae3c7039f5f1af8" candidate_uuid: "32390c92c5f24a8fa2045fe39137363b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475920a79b834257a577f7dec853724a"
I20260620 06:15:29.012563 1634 raft_consensus.cc:3037] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:29.013432 1765 raft_consensus.cc:2454] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32390c92c5f24a8fa2045fe39137363b in term 1.
I20260620 06:15:29.013607 1634 raft_consensus.cc:2454] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32390c92c5f24a8fa2045fe39137363b in term 1.
I20260620 06:15:29.013754 1438 leader_election.cc:302] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32390c92c5f24a8fa2045fe39137363b, 593e207e786d4d4899344b464b2c9483; no voters:
I20260620 06:15:29.013958 1834 raft_consensus.cc:2781] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:29.014137 1834 raft_consensus.cc:686] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 1 LEADER]: Becoming Leader. State: Replica: 32390c92c5f24a8fa2045fe39137363b, State: Running, Role: LEADER
I20260620 06:15:29.014428 1834 consensus_queue.cc:237] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } }
I20260620 06:15:29.017607 1360 catalog_manager.cc:5495] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b reported cstate change: term changed from 0 to 1, leader changed from <none> to 32390c92c5f24a8fa2045fe39137363b (127.14.209.129). New cstate: current_term: 1 leader_uuid: "32390c92c5f24a8fa2045fe39137363b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32390c92c5f24a8fa2045fe39137363b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 41227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 } health_report { overall_health: UNKNOWN } } }
W20260620 06:15:29.033797 1812 tablet.cc:2369] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:29.055789 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.052257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.14.209.129:41227)
0620 06:15:29.052397 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:29.052402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:29.052415 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:15:29.052563 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:29.052566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:29.052625 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:29.052728 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:15:29.052977 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.053468 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.054305 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:29.054525 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:29.054623 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:29.054971 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:15:29.055374 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:15:29.055519 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:29.055628 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 06:15:29.055662 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:15:29.060792 1765 raft_consensus.cc:1261] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 1 FOLLOWER]: Refusing update from remote peer 32390c92c5f24a8fa2045fe39137363b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:29.060804 1634 raft_consensus.cc:1261] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 1 FOLLOWER]: Refusing update from remote peer 32390c92c5f24a8fa2045fe39137363b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:29.061411 1834 consensus_queue.cc:1035] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [LEADER]: Connected to new peer: Peer: permanent_uuid: "593e207e786d4d4899344b464b2c9483" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:29.061594 1839 consensus_queue.cc:1035] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [LEADER]: Connected to new peer: Peer: permanent_uuid: "475920a79b834257a577f7dec853724a" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:29.069298 1842 mvcc.cc:205] Tried to move back new op lower bound from 7298810384627068928 to 7298810384445784064. Current Snapshot: MvccSnapshot[applied={T|T < 7298810384627068928 or (T in {7298810384627068928})}]
I20260620 06:15:29.069487 1843 mvcc.cc:205] Tried to move back new op lower bound from 7298810384627068928 to 7298810384445784064. Current Snapshot: MvccSnapshot[applied={T|T < 7298810384627068928}]
I20260620 06:15:29.075318 1359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:29.075461 1359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:29.077579 1359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 759f861a952e47408580e0a00a06cce5: Sending DeleteTablet for 3 replicas of tablet 6d9b0c863a344f698ae3c7039f5f1af8
I20260620 06:15:29.078276 1744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d9b0c863a344f698ae3c7039f5f1af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:29 UTC) from {username='slave'} at 127.0.0.1:36390
I20260620 06:15:29.078504 1614 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d9b0c863a344f698ae3c7039f5f1af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:29 UTC) from {username='slave'} at 127.0.0.1:54940
I20260620 06:15:29.078742 1483 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d9b0c863a344f698ae3c7039f5f1af8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:29 UTC) from {username='slave'} at 127.0.0.1:59950
I20260620 06:15:29.079541 1852 tablet_replica.cc:330] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: stopping tablet replica
I20260620 06:15:29.079272 1854 tablet_replica.cc:330] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: stopping tablet replica
I20260620 06:15:29.079818 1852 raft_consensus.cc:2229] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:29.080122 1852 raft_consensus.cc:2258] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:29.080269 1854 raft_consensus.cc:2229] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:15:29.081051 1854 raft_consensus.cc:2258] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:29.082261 1852 ts_tablet_manager.cc:1905] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:29.082508 1854 ts_tablet_manager.cc:1905] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:29.085280 1854 ts_tablet_manager.cc:1918] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:29.085357 1854 log.cc:1196] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-0/wal/wals/6d9b0c863a344f698ae3c7039f5f1af8
I20260620 06:15:29.085647 1854 ts_tablet_manager.cc:1939] T 6d9b0c863a344f698ae3c7039f5f1af8 P 32390c92c5f24a8fa2045fe39137363b: Deleting consensus metadata
I20260620 06:15:29.085634 1852 ts_tablet_manager.cc:1918] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:29.085727 1852 log.cc:1196] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/6d9b0c863a344f698ae3c7039f5f1af8
I20260620 06:15:29.085995 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1421
I20260620 06:15:29.086031 1852 ts_tablet_manager.cc:1939] T 6d9b0c863a344f698ae3c7039f5f1af8 P 593e207e786d4d4899344b464b2c9483: Deleting consensus metadata
I20260620 06:15:29.087242 1347 catalog_manager.cc:4841] TS 593e207e786d4d4899344b464b2c9483 (127.14.209.131:42231): tablet 6d9b0c863a344f698ae3c7039f5f1af8 (table test-table [id=c541256979d541fc912e664f40d12c0b]) successfully deleted
I20260620 06:15:29.088637 1853 tablet_replica.cc:330] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: stopping tablet replica
I20260620 06:15:29.088908 1853 raft_consensus.cc:2229] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:29.089162 1853 raft_consensus.cc:2258] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:29.090619 1853 ts_tablet_manager.cc:1905] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:29.093964 1853 ts_tablet_manager.cc:1918] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:29.094061 1853 log.cc:1196] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/6d9b0c863a344f698ae3c7039f5f1af8
I20260620 06:15:29.094380 1853 ts_tablet_manager.cc:1939] T 6d9b0c863a344f698ae3c7039f5f1af8 P 475920a79b834257a577f7dec853724a: Deleting consensus metadata
W20260620 06:15:29.094448 1346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:41227 (error 108)
I20260620 06:15:29.094566 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1552
I20260620 06:15:29.095676 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.095332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:41227 (local address 127.0.0.1:59974)
0620 06:15:29.095481 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:29.095557 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:15:29.095829 1346 catalog_manager.cc:4584] TS 32390c92c5f24a8fa2045fe39137363b (127.14.209.129:41227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d9b0c863a344f698ae3c7039f5f1af8: Network error: Client connection negotiation failed: client connection to 127.14.209.129:41227: connect: Connection refused (error 111)
I20260620 06:15:29.100402 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1683
I20260620 06:15:29.101397 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:29.100517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46401 (local address 127.0.0.1:54956)
0620 06:15:29.101290 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:29.101335 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.130:46401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":709}
I20260620 06:15:29.105656 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1330
2026-06-20T06:15:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:29 dist-test-slave-zt01 krb5kdc[1858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:29 dist-test-slave-zt01 krb5kdc[1858](info): set up 2 sockets
Jun 20 06:15:29 dist-test-slave-zt01 krb5kdc[1858](info): commencing operation
krb5kdc: starting...
W20260620 06:15:30.146939 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:30.269994 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45565
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:30.374641 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:30.374866 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:30.374905 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:30.378743 1871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:30.378799 1871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:30.378819 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:30.378835 1871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:30.378850 1871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:30.383960 1871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:30.384793 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:30.385671 1871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:30.393455 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:30.393491 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:30.393455 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:30.393688 1871 server_base.cc:1034] running on GCE node
I20260620 06:15:30.394279 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:30.395401 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:30.396555 1871 hybrid_clock.cc:648] HybridClock initialized: now 1781936130396532 us; error 35 us; skew 500 ppm
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:30.399215 1871 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:30.399997 1871 webserver.cc:460] Webserver started at http://127.14.209.190:39265/ using document root <none> and password file <none>
I20260620 06:15:30.400512 1871 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:30.400596 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:30.400784 1871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:30.402720 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "f9fd5e3b4682441eba73e41e0b662ce2"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "e6f7e853587dd1220498b2b0e0827da3"
server_key_iv: "178eaf2999d914f310f458e6e22b2434"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.403316 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "f9fd5e3b4682441eba73e41e0b662ce2"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "e6f7e853587dd1220498b2b0e0827da3"
server_key_iv: "178eaf2999d914f310f458e6e22b2434"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.406751 1871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:30.408987 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.410094 1871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:30.410223 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "f9fd5e3b4682441eba73e41e0b662ce2"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "e6f7e853587dd1220498b2b0e0827da3"
server_key_iv: "178eaf2999d914f310f458e6e22b2434"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.410323 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:30.421731 1871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:30.424561 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:30.424758 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:30.431528 1871 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:45565
I20260620 06:15:30.431568 1938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:45565 every 8 connection(s)
I20260620 06:15:30.432415 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:30.434950 1939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:30.435316 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1871
I20260620 06:15:30.435467 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:30.435757 15174 external_mini_cluster.cc:1428] Setting key ccddc2797257fb082eb2989acaa85789
I20260620 06:15:30.441008 1939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2: Bootstrap starting.
I20260620 06:15:30.443137 1939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:30.443957 1939 log.cc:824] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:30.445789 1939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2: No bootstrap required, opened a new log
I20260620 06:15:30.449142 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:30.437025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.14.209.190:45565)
0620 06:15:30.437428 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:30.437444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:30.437474 (+ 30us) server_negotiation.cc:407] Connection header received
0620 06:15:30.438129 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:30.438145 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:30.438433 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:30.438747 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:30.439522 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.440452 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.441316 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.441776 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.441931 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:30.444630 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:30.444634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:30.444645 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:30.444671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:30.446509 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:30.446937 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:30.446941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:30.446945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:30.447112 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:30.447402 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:30.447406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:30.447408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:30.447756 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:30.447968 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:30.448239 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 06:15:30.448495 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260620 06:15:30.449170 1939 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45565 } }
I20260620 06:15:30.449417 1939 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:30.449491 1939 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fd5e3b4682441eba73e41e0b662ce2, State: Initialized, Role: FOLLOWER
I20260620 06:15:30.449869 1939 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45565 } }
I20260620 06:15:30.449980 1939 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:30.450059 1939 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:30.450152 1939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:30.450956 1939 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45565 } }
I20260620 06:15:30.451318 1939 leader_election.cc:302] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9fd5e3b4682441eba73e41e0b662ce2; no voters:
I20260620 06:15:30.451581 1939 leader_election.cc:288] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:30.451691 1944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:30.451944 1944 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [term 1 LEADER]: Becoming Leader. State: Replica: f9fd5e3b4682441eba73e41e0b662ce2, State: Running, Role: LEADER
I20260620 06:15:30.452242 1944 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45565 } }
I20260620 06:15:30.452683 1939 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:30.453758 1945 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45565 } } }
I20260620 06:15:30.453900 1945 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:30.454237 1953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:30.454324 1946 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9fd5e3b4682441eba73e41e0b662ce2. Latest consensus state: current_term: 1 leader_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9fd5e3b4682441eba73e41e0b662ce2" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45565 } } }
I20260620 06:15:30.454490 1946 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:30.457350 1953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:30.461939 1953 catalog_manager.cc:1347] Generated new cluster ID: 2b8e7a34c3af4278976dac11befcb3a1
I20260620 06:15:30.461997 1953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:30.476681 1953 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:30.477521 1953 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:30.486025 1953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2: Generated new TSK 0
I20260620 06:15:30.486681 1953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.209.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
Entry for principal kudu/127.14.209.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab.
I20260620 06:15:30.518302 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45565
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:30.625173 1965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:30.625427 1965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:30.625492 1965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:30.629456 1965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:30.629529 1965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:30.629612 1965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:30.634591 1965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.129.keytab
--principal=kudu/127.14.209.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:30.635540 1965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:30.636507 1965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:30.645578 1971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:30.645560 1970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:30.645635 1973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:30.645886 1965 server_base.cc:1034] running on GCE node
I20260620 06:15:30.646230 1965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:30.646804 1965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:30.647981 1965 hybrid_clock.cc:648] HybridClock initialized: now 1781936130647948 us; error 46 us; skew 500 ppm
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:30.650781 1965 init.cc:377] Logged in from keytab as kudu/127.14.209.129@KRBTEST.COM (short username kudu)
I20260620 06:15:30.651584 1965 webserver.cc:460] Webserver started at http://127.14.209.129:36555/ using document root <none> and password file <none>
I20260620 06:15:30.652066 1965 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:30.652123 1965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:30.652283 1965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:30.654232 1965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "bbedb7591dc447e39d29123ccc284d30"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "0b43d189744613f6fd8c16eac976f1fd"
server_key_iv: "f9d8f9bcc671a423f3a663e81a93a470"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.654788 1965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "bbedb7591dc447e39d29123ccc284d30"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "0b43d189744613f6fd8c16eac976f1fd"
server_key_iv: "f9d8f9bcc671a423f3a663e81a93a470"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.658147 1965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:15:30.660416 1980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.661535 1965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:30.661664 1965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "bbedb7591dc447e39d29123ccc284d30"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "0b43d189744613f6fd8c16eac976f1fd"
server_key_iv: "f9d8f9bcc671a423f3a663e81a93a470"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.661756 1965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:30.670907 1965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:30.673950 1965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:30.674122 1965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:30.674636 1965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:30.675432 1965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:30.675477 1965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.675537 1965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:30.675570 1965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.684096 1965 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:42793
I20260620 06:15:30.684123 2093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:42793 every 8 connection(s)
I20260620 06:15:30.684932 1965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:30.693933 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 1965
I20260620 06:15:30.694085 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:15:30.694408 15174 external_mini_cluster.cc:1428] Setting key 2169fba35e6c39dcd7a63cc0e35cdbd7
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.129@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:30.699329 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:30.687457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:34453 (local address 127.14.209.190:45565)
0620 06:15:30.687664 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:30.687670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:30.688345 (+ 675us) server_negotiation.cc:407] Connection header received
0620 06:15:30.689707 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:30.689710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:30.689758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:30.689833 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:30.690964 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.691575 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.692725 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.692991 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.693081 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:30.695684 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:30.695689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:30.695691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:30.695718 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:30.697431 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:30.697910 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:30.697914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:30.697916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:30.697997 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:30.698319 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:30.698324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:30.698325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:30.698497 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:30.698595 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:30.699036 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 06:15:30.699168 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 06:15:30.700141 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:30.687698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45565 (local address 127.14.209.129:34453)
0620 06:15:30.688194 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:30.688240 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:30.689500 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:30.689996 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:30.690003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:30.690343 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:30.690817 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:30.690830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.691693 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.691696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:30.692578 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:30.692587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.693120 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.693123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:30.693254 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:30.693867 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:30.693889 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:30.695477 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:30.697549 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:30.697554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:30.697568 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:30.697805 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:30.698090 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:30.698093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:30.698095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:30.698226 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:30.698603 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:30.698612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:30.698826 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 06:15:30.698997 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 06:15:30.699319 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":108,"threads_started":1}
I20260620 06:15:30.701228 2094 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45565
I20260620 06:15:30.701463 2094 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:30.701916 2094 heartbeater.cc:511] Master 127.14.209.190:45565 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.209.130@KRBTEST.COM; defaulting to no policy
I20260620 06:15:30.703343 1903 ts_manager.cc:194] Registered new tserver with Master: bbedb7591dc447e39d29123ccc284d30 (127.14.209.129:42793)
I20260620 06:15:30.704406 1903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.129@KRBTEST.COM'} at 127.14.209.129:34453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
Entry for principal kudu/127.14.209.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab.
I20260620 06:15:30.723627 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45565
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:30.829282 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:30.829500 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:30.829540 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:30.833395 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:30.833441 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:30.833531 2099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:30.838452 2099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.130.keytab
--principal=kudu/127.14.209.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:30.839359 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:30.840236 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:30.848855 2105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:30.848901 2104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:30.848949 2107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:30.849045 2099 server_base.cc:1034] running on GCE node
I20260620 06:15:30.849510 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:30.850077 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:30.851250 2099 hybrid_clock.cc:648] HybridClock initialized: now 1781936130851223 us; error 41 us; skew 500 ppm
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:30.853920 2099 init.cc:377] Logged in from keytab as kudu/127.14.209.130@KRBTEST.COM (short username kudu)
I20260620 06:15:30.854677 2099 webserver.cc:460] Webserver started at http://127.14.209.130:44861/ using document root <none> and password file <none>
I20260620 06:15:30.855195 2099 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:30.855257 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:30.855406 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:30.857168 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "b6bd7842d21c402b9c763ccb06f4d36f"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "13a50da90c2601b0506a3de9444b3940"
server_key_iv: "a259e2f0c7d93634144b46539dac6cb6"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.857630 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "b6bd7842d21c402b9c763ccb06f4d36f"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "13a50da90c2601b0506a3de9444b3940"
server_key_iv: "a259e2f0c7d93634144b46539dac6cb6"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.860934 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:30.863026 2114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.864171 2099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:30.864259 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "b6bd7842d21c402b9c763ccb06f4d36f"
format_stamp: "Formatted at 2026-06-20 06:15:30 on dist-test-slave-zt01"
server_key: "13a50da90c2601b0506a3de9444b3940"
server_key_iv: "a259e2f0c7d93634144b46539dac6cb6"
server_key_version: "encryptionkey@0"
I20260620 06:15:30.864334 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:30.877311 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:30.879801 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:30.879945 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:30.880388 2099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:30.881089 2099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:30.881134 2099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.881170 2099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:30.881191 2099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:30.889534 2099 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:40323
I20260620 06:15:30.889568 2227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:40323 every 8 connection(s)
I20260620 06:15:30.890311 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:30.899722 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2099
I20260620 06:15:30.899884 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
Jun 20 06:15:30 dist-test-slave-zt01 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936130, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.130@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:30.900225 15174 external_mini_cluster.cc:1428] Setting key 398f2783260c2b9a7a4017c36e61136a
I20260620 06:15:30.905083 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:30.892363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:43569 (local address 127.14.209.190:45565)
0620 06:15:30.892503 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:30.892508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:30.893335 (+ 827us) server_negotiation.cc:407] Connection header received
0620 06:15:30.894637 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:30.894640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:30.894685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:30.894764 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:30.895978 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.896586 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.897795 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.898044 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.898145 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:30.900710 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:30.900714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:30.900717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:30.900750 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:30.902331 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:30.903186 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:30.903191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:30.903193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:30.903352 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:30.903825 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:30.903830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:30.903833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:30.904111 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:30.904233 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:30.904710 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 06:15:30.904846 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:15:30.905604 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:30.892731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45565 (local address 127.14.209.130:43569)
0620 06:15:30.893193 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:30.893232 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:30.894416 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:30.894930 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:30.894936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:30.895287 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:30.895718 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:30.895731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.896745 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.896748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:30.897648 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:30.897657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:30.898180 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:30.898182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:30.898315 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:30.898863 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:30.898879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:30.900520 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:30.902457 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:30.902466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:30.902482 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:30.902850 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:30.903524 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:30.903530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:30.903534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:30.903726 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:30.904234 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:30.904239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:30.904447 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 06:15:30.904594 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 06:15:30.904814 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
I20260620 06:15:30.906646 2228 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45565
I20260620 06:15:30.906867 2228 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:30.907377 2228 heartbeater.cc:511] Master 127.14.209.190:45565 requested a full tablet report, sending...
I20260620 06:15:30.908346 1903 ts_manager.cc:194] Registered new tserver with Master: b6bd7842d21c402b9c763ccb06f4d36f (127.14.209.130:40323)
I20260620 06:15:30.908901 1903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.130@KRBTEST.COM'} at 127.14.209.130:43569
WARNING: no policy specified for kudu/127.14.209.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
Entry for principal kudu/127.14.209.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab.
I20260620 06:15:30.926160 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45565
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:31.030974 2233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:31.031278 2233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:31.031359 2233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:31.035347 2233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:31.035421 2233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:31.035512 2233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:31.040512 2233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.131.keytab
--principal=kudu/127.14.209.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:31.041450 2233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:31.042371 2233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:31.051180 2238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.051185 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.051219 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:31.051342 2233 server_base.cc:1034] running on GCE node
I20260620 06:15:31.051749 2233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:31.052317 2233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:31.053488 2233 hybrid_clock.cc:648] HybridClock initialized: now 1781936131053445 us; error 47 us; skew 500 ppm
Jun 20 06:15:31 dist-test-slave-zt01 krb5kdc[1858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936131, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:31.056154 2233 init.cc:377] Logged in from keytab as kudu/127.14.209.131@KRBTEST.COM (short username kudu)
I20260620 06:15:31.056957 2233 webserver.cc:460] Webserver started at http://127.14.209.131:34473/ using document root <none> and password file <none>
I20260620 06:15:31.057458 2233 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:31.057538 2233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:31.057739 2233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:31.059625 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "dd949e409b8048bab89357dced9ee0e6"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "b778b3def62f94b880336d909ad136ae"
server_key_iv: "dab54ff6b09c9f2359013d8971748e12"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.060137 2233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "dd949e409b8048bab89357dced9ee0e6"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "b778b3def62f94b880336d909ad136ae"
server_key_iv: "dab54ff6b09c9f2359013d8971748e12"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.063457 2233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:15:31.065593 2248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.066694 2233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:31.066805 2233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "dd949e409b8048bab89357dced9ee0e6"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "b778b3def62f94b880336d909ad136ae"
server_key_iv: "dab54ff6b09c9f2359013d8971748e12"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.066897 2233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:31.077556 2233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:31.080164 2233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:31.080336 2233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:31.080822 2233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:31.081562 2233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:31.081606 2233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.081663 2233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:31.081701 2233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.089874 2233 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:43645
I20260620 06:15:31.089923 2361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:43645 every 8 connection(s)
I20260620 06:15:31.090644 2233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:31.091343 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2233
I20260620 06:15:31.091487 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:15:31.091720 15174 external_mini_cluster.cc:1428] Setting key 9d5299f4dc05be92aa1947bab0fb1c84
Jun 20 06:15:31 dist-test-slave-zt01 krb5kdc[1858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936131, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.131@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:31.103237 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.092388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:33923 (local address 127.14.209.190:45565)
0620 06:15:31.092519 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.092522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.093502 (+ 980us) server_negotiation.cc:407] Connection header received
0620 06:15:31.094858 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.094862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.094917 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.094992 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:31.096058 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.096584 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.097702 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.097922 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.098028 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.100333 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:31.100336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:31.100338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:31.100359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:31.101508 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:31.101965 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:31.101967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:31.101968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:31.102032 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:31.102315 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:31.102317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:31.102318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:31.102465 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:31.102535 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.102973 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.103112 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:15:31.103935 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.092788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:45565 (local address 127.14.209.131:33923)
0620 06:15:31.093373 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.093411 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.094672 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.095178 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.095185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.095480 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:31.095922 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.095933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.096732 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.096737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.097581 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.097589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.098059 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.098063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.098233 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.098768 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:31.098783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:31.100192 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:31.101638 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:31.101643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:31.101652 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:31.101883 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:31.102106 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:31.102110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:31.102112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:31.102243 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:31.102541 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:31.102546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:31.102780 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.102927 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.103200 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":133,"threads_started":1}
I20260620 06:15:31.105067 2362 heartbeater.cc:347] Connected to a master server at 127.14.209.190:45565
I20260620 06:15:31.105294 2362 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:31.105705 2362 heartbeater.cc:511] Master 127.14.209.190:45565 requested a full tablet report, sending...
I20260620 06:15:31.106658 1902 ts_manager.cc:194] Registered new tserver with Master: dd949e409b8048bab89357dced9ee0e6 (127.14.209.131:43645)
I20260620 06:15:31.107291 1902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.209.131@KRBTEST.COM'} at 127.14.209.131:33923
I20260620 06:15:31.116899 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:15:31.125134 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.118569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.14.209.190:45565)
0620 06:15:31.118680 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.118684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.118868 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:15:31.118961 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.118963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.118995 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.119112 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:31.119729 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.120191 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.120965 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.121166 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.121241 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.122431 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:31.122434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:31.122436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:31.122457 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:31.123875 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:31.124216 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:31.124218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:31.124219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:31.124285 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:31.124494 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:31.124496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:31.124497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:31.124676 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:31.124773 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.124951 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.125030 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:15:31.127811 1902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:15:31.130684 1902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:15:31.144418 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.138905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.14.209.131:43645)
0620 06:15:31.139631 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.139643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.139667 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:15:31.139740 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.139745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.139953 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.140474 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:31.140592 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.141578 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.142936 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.143785 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.143908 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.144027 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.144198 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.144284 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":164,"threads_started":1}
I20260620 06:15:31.144414 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.138818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:43645 (local address 127.0.0.1:38304)
0620 06:15:31.139163 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.139206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.139316 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.140151 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.140157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.140180 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:31.140323 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.140329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.141691 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.141698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.142802 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.142812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.143895 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.143898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.143996 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.144029 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.144168 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.144278 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"spinlock_wait_cycles":6272,"thread_start_us":91,"threads_started":1}
I20260620 06:15:31.144989 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.138887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:40323 (local address 127.0.0.1:45680)
0620 06:15:31.139562 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.139575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.139654 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.140943 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.140945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.140962 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:31.141058 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.141063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.142380 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.142386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.143458 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.143466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.144554 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.144557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.144711 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.144734 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.144797 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.144858 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":20,"thread_start_us":87,"threads_started":1}
I20260620 06:15:31.145134 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.139579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.14.209.130:40323)
0620 06:15:31.140500 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.140512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.140534 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:15:31.140608 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.140613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.140781 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.140960 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:31.141172 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.142274 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.143578 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.144407 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.144732 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.144856 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.144900 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.144979 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":101,"threads_started":1}
I20260620 06:15:31.145439 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.139631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:42793 (local address 127.0.0.1:53622)
0620 06:15:31.140109 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.140132 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.140250 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.140767 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.140770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.140786 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:31.140918 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.140925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.142349 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.142352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.143581 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.143591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.144554 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.144557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.145217 (+ 660us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.145243 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.145301 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.145348 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":79,"spinlock_wait_cycles":384,"thread_start_us":87,"threads_started":1}
I20260620 06:15:31.146090 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.139962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.14.209.129:42793)
0620 06:15:31.140346 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.140359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.140382 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:15:31.140452 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.140456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.140617 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.140779 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:31.141044 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.142192 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.143695 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.144439 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.145704 (+ 1265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.145808 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.145856 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.145943 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":113,"threads_started":1}
I20260620 06:15:31.147500 2162 tablet_service.cc:1467] Processing CreateTablet for tablet 3d137b3d1c6a453389a30f406b18ed58 (DEFAULT_TABLE table=test-table [id=e4040b7748d04ebd9baf640aaccdc879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:31.147500 2296 tablet_service.cc:1467] Processing CreateTablet for tablet 3d137b3d1c6a453389a30f406b18ed58 (DEFAULT_TABLE table=test-table [id=e4040b7748d04ebd9baf640aaccdc879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:31.148540 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d137b3d1c6a453389a30f406b18ed58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:31.148540 2296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d137b3d1c6a453389a30f406b18ed58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:31.148995 2028 tablet_service.cc:1467] Processing CreateTablet for tablet 3d137b3d1c6a453389a30f406b18ed58 (DEFAULT_TABLE table=test-table [id=e4040b7748d04ebd9baf640aaccdc879]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:31.149821 2028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d137b3d1c6a453389a30f406b18ed58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:31.156029 2379 tablet_bootstrap.cc:492] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: Bootstrap starting.
I20260620 06:15:31.156802 2381 tablet_bootstrap.cc:492] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Bootstrap starting.
I20260620 06:15:31.157044 2380 tablet_bootstrap.cc:492] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Bootstrap starting.
I20260620 06:15:31.158274 2379 tablet_bootstrap.cc:654] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:31.158756 2380 tablet_bootstrap.cc:654] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:31.158756 2381 tablet_bootstrap.cc:654] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:31.159485 2379 log.cc:824] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:31.160020 2380 log.cc:824] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:31.160020 2381 log.cc:824] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:31.161654 2379 tablet_bootstrap.cc:492] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: No bootstrap required, opened a new log
I20260620 06:15:31.161665 2381 tablet_bootstrap.cc:492] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: No bootstrap required, opened a new log
I20260620 06:15:31.161841 2381 ts_tablet_manager.cc:1397] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 06:15:31.161837 2379 ts_tablet_manager.cc:1397] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:15:31.162184 2380 tablet_bootstrap.cc:492] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: No bootstrap required, opened a new log
I20260620 06:15:31.162345 2380 ts_tablet_manager.cc:1397] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:15:31.164886 2381 raft_consensus.cc:348] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.165117 2381 raft_consensus.cc:374] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:31.165171 2381 raft_consensus.cc:729] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bd7842d21c402b9c763ccb06f4d36f, State: Initialized, Role: FOLLOWER
I20260620 06:15:31.165582 2381 consensus_queue.cc:260] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.165562 2379 raft_consensus.cc:348] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.165789 2379 raft_consensus.cc:374] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:31.165843 2379 raft_consensus.cc:729] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbedb7591dc447e39d29123ccc284d30, State: Initialized, Role: FOLLOWER
I20260620 06:15:31.165913 2380 raft_consensus.cc:348] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.166150 2380 raft_consensus.cc:374] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:31.166229 2380 raft_consensus.cc:729] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd949e409b8048bab89357dced9ee0e6, State: Initialized, Role: FOLLOWER
I20260620 06:15:31.166252 2228 heartbeater.cc:503] Master 127.14.209.190:45565 was elected leader, sending a full tablet report...
I20260620 06:15:31.166565 2381 ts_tablet_manager.cc:1428] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 06:15:31.166610 2380 consensus_queue.cc:260] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.167392 2379 consensus_queue.cc:260] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.167646 2362 heartbeater.cc:503] Master 127.14.209.190:45565 was elected leader, sending a full tablet report...
I20260620 06:15:31.167729 2380 ts_tablet_manager.cc:1428] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:15:31.168164 2379 ts_tablet_manager.cc:1428] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:15:31.168426 2094 heartbeater.cc:503] Master 127.14.209.190:45565 was elected leader, sending a full tablet report...
W20260620 06:15:31.186010 2095 tablet.cc:2369] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:31.262064 2385 raft_consensus.cc:482] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:31.262293 2385 raft_consensus.cc:504] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.263404 2385 leader_election.cc:288] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd949e409b8048bab89357dced9ee0e6 (127.14.209.131:43645), bbedb7591dc447e39d29123ccc284d30 (127.14.209.129:42793)
I20260620 06:15:31.267230 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.263853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:40291 (local address 127.14.209.129:42793)
0620 06:15:31.263964 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.263969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.263988 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:15:31.264032 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.264034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.264073 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.264162 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:31.264495 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.265104 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.266256 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.266644 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.266769 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.266822 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.267012 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.267103 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:15:31.267232 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.263640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:42793 (local address 127.14.209.130:40291)
0620 06:15:31.263751 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.263767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.263852 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.264194 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.264198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.264217 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:31.264370 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.264377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.265204 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.265210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.266136 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.266145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.266760 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.266764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.266870 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.266896 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.266992 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.267099 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1,"spinlock_wait_cycles":29696}
I20260620 06:15:31.267478 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.263734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:58613 (local address 127.14.209.131:43645)
0620 06:15:31.263877 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.263882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.264215 (+ 333us) server_negotiation.cc:407] Connection header received
0620 06:15:31.264400 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.264404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.264463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.264551 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:31.264841 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.265345 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.266456 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.266871 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.266988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.267037 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.267307 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.267373 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:15:31.267495 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.263640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:43645 (local address 127.14.209.130:58613)
0620 06:15:31.264124 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.264135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.264275 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.264569 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.264572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.264599 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:31.264737 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.264743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.265448 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.265454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.266341 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.266350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.266988 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.266992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.267172 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.267198 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.267292 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.267356 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":188,"spinlock_wait_cycles":768,"thread_start_us":138,"threads_started":1}
I20260620 06:15:31.267930 2048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137b3d1c6a453389a30f406b18ed58" candidate_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbedb7591dc447e39d29123ccc284d30" is_pre_election: true
I20260620 06:15:31.268067 2316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137b3d1c6a453389a30f406b18ed58" candidate_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd949e409b8048bab89357dced9ee0e6" is_pre_election: true
I20260620 06:15:31.268244 2048 raft_consensus.cc:2454] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6bd7842d21c402b9c763ccb06f4d36f in term 0.
I20260620 06:15:31.268386 2316 raft_consensus.cc:2454] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6bd7842d21c402b9c763ccb06f4d36f in term 0.
I20260620 06:15:31.268729 2115 leader_election.cc:302] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6bd7842d21c402b9c763ccb06f4d36f, bbedb7591dc447e39d29123ccc284d30; no voters:
I20260620 06:15:31.268978 2385 raft_consensus.cc:2781] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:15:31.269057 2385 raft_consensus.cc:482] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:15:31.269100 2385 raft_consensus.cc:3037] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:31.270147 2385 raft_consensus.cc:504] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.270444 2385 leader_election.cc:288] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [CANDIDATE]: Term 1 election: Requested vote from peers dd949e409b8048bab89357dced9ee0e6 (127.14.209.131:43645), bbedb7591dc447e39d29123ccc284d30 (127.14.209.129:42793)
I20260620 06:15:31.270787 2316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137b3d1c6a453389a30f406b18ed58" candidate_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd949e409b8048bab89357dced9ee0e6"
I20260620 06:15:31.270844 2048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d137b3d1c6a453389a30f406b18ed58" candidate_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbedb7591dc447e39d29123ccc284d30"
I20260620 06:15:31.270942 2048 raft_consensus.cc:3037] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:31.270941 2316 raft_consensus.cc:3037] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:31.272171 2316 raft_consensus.cc:2454] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6bd7842d21c402b9c763ccb06f4d36f in term 1.
I20260620 06:15:31.272171 2048 raft_consensus.cc:2454] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6bd7842d21c402b9c763ccb06f4d36f in term 1.
I20260620 06:15:31.272504 2116 leader_election.cc:302] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b6bd7842d21c402b9c763ccb06f4d36f, bbedb7591dc447e39d29123ccc284d30, dd949e409b8048bab89357dced9ee0e6; no voters:
I20260620 06:15:31.272670 2385 raft_consensus.cc:2781] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:31.272918 2385 raft_consensus.cc:686] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 1 LEADER]: Becoming Leader. State: Replica: b6bd7842d21c402b9c763ccb06f4d36f, State: Running, Role: LEADER
I20260620 06:15:31.273236 2385 consensus_queue.cc:237] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } }
I20260620 06:15:31.276101 1903 catalog_manager.cc:5495] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6bd7842d21c402b9c763ccb06f4d36f (127.14.209.130). New cstate: current_term: 1 leader_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6bd7842d21c402b9c763ccb06f4d36f" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 40323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 } health_report { overall_health: UNKNOWN } } }
I20260620 06:15:31.296017 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.292972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.14.209.130:40323)
0620 06:15:31.293078 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.293082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.293094 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:15:31.293182 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.293184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.293230 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.293313 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:15:31.293521 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.293988 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.294754 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.294953 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.295028 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.295312 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:15:31.295674 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:15:31.295761 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.295883 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.295929 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:15:31.301172 2048 raft_consensus.cc:1261] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 1 FOLLOWER]: Refusing update from remote peer b6bd7842d21c402b9c763ccb06f4d36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:31.301177 2316 raft_consensus.cc:1261] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 1 FOLLOWER]: Refusing update from remote peer b6bd7842d21c402b9c763ccb06f4d36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:31.301779 2385 consensus_queue.cc:1035] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbedb7591dc447e39d29123ccc284d30" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:31.301928 2389 consensus_queue.cc:1035] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd949e409b8048bab89357dced9ee0e6" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:31.309491 2394 mvcc.cc:205] Tried to move back new op lower bound from 7298810393803235328 to 7298810393697378304. Current Snapshot: MvccSnapshot[applied={T|T < 7298810393803235328}]
I20260620 06:15:31.310540 2393 mvcc.cc:205] Tried to move back new op lower bound from 7298810393803235328 to 7298810393697378304. Current Snapshot: MvccSnapshot[applied={T|T < 7298810393803235328}]
I20260620 06:15:31.316812 1902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:31.317257 1902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:31.319551 1902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9fd5e3b4682441eba73e41e0b662ce2: Sending DeleteTablet for 3 replicas of tablet 3d137b3d1c6a453389a30f406b18ed58
I20260620 06:15:31.320209 2296 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137b3d1c6a453389a30f406b18ed58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:31 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:38304
I20260620 06:15:31.320351 2028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137b3d1c6a453389a30f406b18ed58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:31 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:53622
I20260620 06:15:31.320497 2162 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d137b3d1c6a453389a30f406b18ed58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:31 UTC) from {username='kudu', principal='kudu/127.14.209.190@KRBTEST.COM'} at 127.0.0.1:45680
I20260620 06:15:31.320794 2403 tablet_replica.cc:330] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30: stopping tablet replica
I20260620 06:15:31.320919 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1965
I20260620 06:15:31.321173 2403 raft_consensus.cc:2229] T 3d137b3d1c6a453389a30f406b18ed58 P bbedb7591dc447e39d29123ccc284d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:31.321600 2404 tablet_replica.cc:330] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: stopping tablet replica
I20260620 06:15:31.321966 2404 raft_consensus.cc:2229] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:15:31.322374 2404 raft_consensus.cc:2258] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:31.323235 2402 tablet_replica.cc:330] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: stopping tablet replica
I20260620 06:15:31.323490 2402 raft_consensus.cc:2229] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:31.323758 2402 raft_consensus.cc:2258] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:31.325103 2404 ts_tablet_manager.cc:1905] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:31.325215 2402 ts_tablet_manager.cc:1905] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:15:31.328029 1887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:42793 (error 108)
I20260620 06:15:31.328620 2404 ts_tablet_manager.cc:1918] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:31.328619 2402 ts_tablet_manager.cc:1918] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:31.328753 2402 log.cc:1196] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/3d137b3d1c6a453389a30f406b18ed58
I20260620 06:15:31.328847 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2099
I20260620 06:15:31.328871 2404 log.cc:1196] T 3d137b3d1c6a453389a30f406b18ed58 P b6bd7842d21c402b9c763ccb06f4d36f: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935885709390-15174-0/minicluster-data/ts-1/wal/wals/3d137b3d1c6a453389a30f406b18ed58
I20260620 06:15:31.330529 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.330257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:42793 (local address 127.0.0.1:53630)
0620 06:15:31.330387 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.330460 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 06:15:31.330708 1887 catalog_manager.cc:4584] TS bbedb7591dc447e39d29123ccc284d30 (127.14.209.129:42793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d137b3d1c6a453389a30f406b18ed58: Network error: Client connection negotiation failed: client connection to 127.14.209.129:42793: connect: Connection refused (error 111)
I20260620 06:15:31.331341 2402 ts_tablet_manager.cc:1939] T 3d137b3d1c6a453389a30f406b18ed58 P dd949e409b8048bab89357dced9ee0e6: Deleting consensus metadata
I20260620 06:15:31.332553 1888 catalog_manager.cc:4841] TS dd949e409b8048bab89357dced9ee0e6 (127.14.209.131:43645): tablet 3d137b3d1c6a453389a30f406b18ed58 (table test-table [id=e4040b7748d04ebd9baf640aaccdc879]) successfully deleted
I20260620 06:15:31.336417 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2233
I20260620 06:15:31.336997 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.336677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:40323 (local address 127.0.0.1:45698)
0620 06:15:31.336855 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.336910 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.130:40323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 06:15:31.342016 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 1871
2026-06-20T06:15:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T06:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:31Z Disabled control of system clock
I20260620 06:15:31.372836 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:43295
--webserver_interface=127.14.209.190
--webserver_port=0
--builtin_ntp_servers=127.14.209.148:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:15:31.476454 2409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:31.476720 2409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:31.476778 2409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:31.481110 2409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:31.481161 2409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:31.481177 2409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:31.481194 2409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:31.481211 2409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:31.485968 2409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:43295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:31.486738 2409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:31.487957 2409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:31.495445 2417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.495469 2415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.495453 2414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:31.496438 2409 server_base.cc:1034] running on GCE node
I20260620 06:15:31.496948 2409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:31.497938 2409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:31.499140 2409 hybrid_clock.cc:648] HybridClock initialized: now 1781936131499120 us; error 31 us; skew 500 ppm
I20260620 06:15:31.500818 2409 webserver.cc:460] Webserver started at http://127.14.209.190:36211/ using document root <none> and password file <none>
I20260620 06:15:31.501348 2409 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:31.501431 2409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:31.501642 2409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:31.503513 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "c247656513d2412a8ff88e09149464d1"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "f2fdb559cdb32ef2f6f5e8c8762dde72"
server_key_iv: "c16c9aa1a14ec0f84c0945ce5aae66d9"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.504032 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "c247656513d2412a8ff88e09149464d1"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "f2fdb559cdb32ef2f6f5e8c8762dde72"
server_key_iv: "c16c9aa1a14ec0f84c0945ce5aae66d9"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.507362 2409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:31.509466 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.510532 2409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:31.510691 2409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "c247656513d2412a8ff88e09149464d1"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "f2fdb559cdb32ef2f6f5e8c8762dde72"
server_key_iv: "c16c9aa1a14ec0f84c0945ce5aae66d9"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.510793 2409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:31.560753 2409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:31.561478 2409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:31.561661 2409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:31.568208 2409 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:43295
I20260620 06:15:31.568267 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:43295 every 8 connection(s)
I20260620 06:15:31.569067 2409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:31.571969 2476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:31.576974 2476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1: Bootstrap starting.
I20260620 06:15:31.578192 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2409
I20260620 06:15:31.578326 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:31.578600 15174 external_mini_cluster.cc:1428] Setting key d8d79f73e79904d8dcdfc2e25c07f458
I20260620 06:15:31.579308 2476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:31.580087 2476 log.cc:824] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:31.582160 2476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1: No bootstrap required, opened a new log
I20260620 06:15:31.586233 2476 raft_consensus.cc:348] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247656513d2412a8ff88e09149464d1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43295 } }
I20260620 06:15:31.586483 2476 raft_consensus.cc:374] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:31.586575 2476 raft_consensus.cc:729] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c247656513d2412a8ff88e09149464d1, State: Initialized, Role: FOLLOWER
I20260620 06:15:31.587037 2476 consensus_queue.cc:260] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247656513d2412a8ff88e09149464d1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43295 } }
I20260620 06:15:31.587212 2476 raft_consensus.cc:388] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:31.587277 2476 raft_consensus.cc:482] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:31.587384 2476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:31.588397 2476 raft_consensus.cc:504] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247656513d2412a8ff88e09149464d1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43295 } }
I20260620 06:15:31.588718 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.580021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.14.209.190:43295)
0620 06:15:31.580640 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.580654 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.580689 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:15:31.581449 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.581472 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.581846 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.582191 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:31.582621 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.584085 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.584996 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.585773 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.585930 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.586719 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:31.586725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:31.586739 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:31.587222 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:31.587296 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:31.587323 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:31.587452 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.587597 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.587839 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":154,"threads_started":1}
I20260620 06:15:31.588730 2476 leader_election.cc:302] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c247656513d2412a8ff88e09149464d1; no voters:
I20260620 06:15:31.589052 2476 leader_election.cc:288] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:31.589162 2481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:31.589346 2481 raft_consensus.cc:686] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [term 1 LEADER]: Becoming Leader. State: Replica: c247656513d2412a8ff88e09149464d1, State: Running, Role: LEADER
I20260620 06:15:31.589618 2481 consensus_queue.cc:237] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247656513d2412a8ff88e09149464d1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43295 } }
I20260620 06:15:31.590190 2476 sys_catalog.cc:564] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:31.591315 2483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c247656513d2412a8ff88e09149464d1. Latest consensus state: current_term: 1 leader_uuid: "c247656513d2412a8ff88e09149464d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247656513d2412a8ff88e09149464d1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43295 } } }
I20260620 06:15:31.591440 2483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:31.591717 2482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c247656513d2412a8ff88e09149464d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c247656513d2412a8ff88e09149464d1" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 43295 } } }
I20260620 06:15:31.591830 2482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:31.592058 2490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:31.594496 2490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:31.599089 2490 catalog_manager.cc:1347] Generated new cluster ID: bfb00735a85042739e191985bc1b83ab
I20260620 06:15:31.599149 2490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:31.611117 2490 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:31.612161 2490 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:31.617532 2490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1: Generated new TSK 0
I20260620 06:15:31.618144 2490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:31.625309 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.129:0
--local_ip_for_outbound_sockets=127.14.209.129
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43295
--builtin_ntp_servers=127.14.209.148:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:15:31.728372 2500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:31.728582 2500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:31.728648 2500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:31.732517 2500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:31.732558 2500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:31.732609 2500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.129
I20260620 06:15:31.737134 2500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.209.129
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.129
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:31.738005 2500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:31.739223 2500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:31.747975 2505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.747931 2506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.747927 2508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:31.747991 2500 server_base.cc:1034] running on GCE node
I20260620 06:15:31.748469 2500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:31.749074 2500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:31.750248 2500 hybrid_clock.cc:648] HybridClock initialized: now 1781936131750221 us; error 41 us; skew 500 ppm
I20260620 06:15:31.752028 2500 webserver.cc:460] Webserver started at http://127.14.209.129:37029/ using document root <none> and password file <none>
I20260620 06:15:31.752560 2500 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:31.752646 2500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:31.752862 2500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:31.754729 2500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data/instance:
uuid: "129faaecf6ce4172b17eba3852fa040b"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "e04428ab472f0d67f056b8889ca832b6"
server_key_iv: "1bb8ddc224dafae926acc5deee62cf15"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.755313 2500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance:
uuid: "129faaecf6ce4172b17eba3852fa040b"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "e04428ab472f0d67f056b8889ca832b6"
server_key_iv: "1bb8ddc224dafae926acc5deee62cf15"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.758543 2500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 06:15:31.760857 2514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.761906 2500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:31.762030 2500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal
uuid: "129faaecf6ce4172b17eba3852fa040b"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "e04428ab472f0d67f056b8889ca832b6"
server_key_iv: "1bb8ddc224dafae926acc5deee62cf15"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.762130 2500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:31.781718 2500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:31.782269 2500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:31.782446 2500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:31.782946 2500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:31.783820 2500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:31.783887 2500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.783949 2500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:31.783989 2500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.792454 2500 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.129:44349
I20260620 06:15:31.792529 2627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.129:44349 every 8 connection(s)
I20260620 06:15:31.793243 2500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/data/info.pb
I20260620 06:15:31.800943 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2500
I20260620 06:15:31.801066 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-0/wal/instance
I20260620 06:15:31.801404 15174 external_mini_cluster.cc:1428] Setting key ca6e02816d05274dda7c92a2b682189c
I20260620 06:15:31.801822 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.794825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:55269 (local address 127.14.209.190:43295)
0620 06:15:31.794947 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.794951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.795934 (+ 983us) server_negotiation.cc:407] Connection header received
0620 06:15:31.797238 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.797241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.797285 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.797360 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:31.798118 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.798668 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.799950 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.800202 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.800292 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.801357 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:31.801360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:31.801362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:31.801373 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:31.801391 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:31.801398 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:31.801486 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.801641 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.801692 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:15:31.802760 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.795231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:43295 (local address 127.14.209.129:55269)
0620 06:15:31.795807 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.795843 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.797007 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.797496 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.797501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.797573 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:31.797984 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.797995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.798794 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.798797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.799824 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.799834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.800351 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.800354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.800492 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.801081 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:31.801084 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:31.801133 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:31.801207 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:31.801490 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:31.801500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:31.801506 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.801633 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.801891 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
I20260620 06:15:31.803474 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.130:0
--local_ip_for_outbound_sockets=127.14.209.130
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43295
--builtin_ntp_servers=127.14.209.148:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:15:31.803876 2628 heartbeater.cc:347] Connected to a master server at 127.14.209.190:43295
I20260620 06:15:31.804093 2628 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:31.804518 2628 heartbeater.cc:511] Master 127.14.209.190:43295 requested a full tablet report, sending...
I20260620 06:15:31.805804 2440 ts_manager.cc:194] Registered new tserver with Master: 129faaecf6ce4172b17eba3852fa040b (127.14.209.129:44349)
I20260620 06:15:31.806954 2440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.129:55269
W20260620 06:15:31.906700 2631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:31.906908 2631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:31.906944 2631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:31.910863 2631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:31.910908 2631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:31.910990 2631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.130
I20260620 06:15:31.916175 2631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.209.130
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.130
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:31.917012 2631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:31.918171 2631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:15:31.926781 2631 server_base.cc:1034] running on GCE node
W20260620 06:15:31.926751 2636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.926726 2637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:31.926738 2639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:31.927341 2631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:31.927892 2631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:31.929077 2631 hybrid_clock.cc:648] HybridClock initialized: now 1781936131929045 us; error 41 us; skew 500 ppm
I20260620 06:15:31.930786 2631 webserver.cc:460] Webserver started at http://127.14.209.130:41825/ using document root <none> and password file <none>
I20260620 06:15:31.931378 2631 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:31.931461 2631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:31.931669 2631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:31.933513 2631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data/instance:
uuid: "3a19d5523eca466c8459d148191b7377"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "fb286875803ffa60073ec6fd9d1fc0d5"
server_key_iv: "ef6116193de0b24019e5a55f370029da"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.934026 2631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance:
uuid: "3a19d5523eca466c8459d148191b7377"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "fb286875803ffa60073ec6fd9d1fc0d5"
server_key_iv: "ef6116193de0b24019e5a55f370029da"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.937354 2631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:15:31.939553 2645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.940606 2631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:31.940732 2631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal
uuid: "3a19d5523eca466c8459d148191b7377"
format_stamp: "Formatted at 2026-06-20 06:15:31 on dist-test-slave-zt01"
server_key: "fb286875803ffa60073ec6fd9d1fc0d5"
server_key_iv: "ef6116193de0b24019e5a55f370029da"
server_key_version: "encryptionkey@0"
I20260620 06:15:31.940828 2631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:31.950434 2631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:31.950953 2631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:31.951155 2631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:31.951642 2631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:31.952440 2631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:31.952512 2631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.952575 2631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:31.952620 2631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:31.960975 2631 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.130:46009
I20260620 06:15:31.961014 2758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.130:46009 every 8 connection(s)
I20260620 06:15:31.961714 2631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/data/info.pb
I20260620 06:15:31.968811 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2631
I20260620 06:15:31.968946 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-1/wal/instance
I20260620 06:15:31.969250 15174 external_mini_cluster.cc:1428] Setting key d102425faa15d04a2d14ecd7b735eaff
I20260620 06:15:31.970213 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.963274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.130:60499 (local address 127.14.209.190:43295)
0620 06:15:31.963415 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:31.963420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:31.964368 (+ 948us) server_negotiation.cc:407] Connection header received
0620 06:15:31.965668 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:31.965671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:31.965716 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:31.965791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:31.966601 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.967206 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.968255 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.968462 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.968544 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.969742 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:31.969746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:31.969749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:31.969761 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:31.969786 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:31.969794 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:31.969866 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:31.970019 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 06:15:31.970076 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:15:31.970978 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:31.963643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:43295 (local address 127.14.209.130:60499)
0620 06:15:31.964246 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:31.964280 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:31.965491 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:31.965964 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:31.965975 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:31.966064 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:31.966485 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.966496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.967316 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.967318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.968147 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:31.968156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:31.968657 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:31.968662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:31.968859 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:31.969463 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:31.969466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:31.969512 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:31.969581 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:31.969880 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:31.969889 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:31.969894 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:15:31.970005 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:15:31.970243 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260620 06:15:31.971043 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.209.131:0
--local_ip_for_outbound_sockets=127.14.209.131
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43295
--builtin_ntp_servers=127.14.209.148:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:15:31.971988 2759 heartbeater.cc:347] Connected to a master server at 127.14.209.190:43295
I20260620 06:15:31.972200 2759 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:31.972697 2759 heartbeater.cc:511] Master 127.14.209.190:43295 requested a full tablet report, sending...
I20260620 06:15:31.973608 2440 ts_manager.cc:194] Registered new tserver with Master: 3a19d5523eca466c8459d148191b7377 (127.14.209.130:46009)
I20260620 06:15:31.974170 2440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.130:60499
W20260620 06:15:32.074496 2762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:32.074728 2762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:32.074806 2762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:32.078802 2762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:32.078868 2762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:32.078951 2762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.209.131
I20260620 06:15:32.083464 2762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.209.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.209.131
--webserver_port=0
--tserver_master_addrs=127.14.209.190:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.209.131
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:32.084311 2762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:32.085562 2762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:32.094740 2768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:32.094833 2762 server_base.cc:1034] running on GCE node
W20260620 06:15:32.094743 2767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:32.094779 2770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:32.095430 2762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:32.096184 2762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:32.097432 2762 hybrid_clock.cc:648] HybridClock initialized: now 1781936132097419 us; error 24 us; skew 500 ppm
I20260620 06:15:32.099237 2762 webserver.cc:460] Webserver started at http://127.14.209.131:41061/ using document root <none> and password file <none>
I20260620 06:15:32.099786 2762 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:32.099875 2762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:32.100086 2762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:32.102774 2762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data/instance:
uuid: "342dc8d2e3304a08adf63e3d23dd7ac8"
format_stamp: "Formatted at 2026-06-20 06:15:32 on dist-test-slave-zt01"
server_key: "665dfa1a5cccb415a094404b88fbfd57"
server_key_iv: "365f7ea30c5886a8b052fdfd5e0b62a2"
server_key_version: "encryptionkey@0"
I20260620 06:15:32.103469 2762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance:
uuid: "342dc8d2e3304a08adf63e3d23dd7ac8"
format_stamp: "Formatted at 2026-06-20 06:15:32 on dist-test-slave-zt01"
server_key: "665dfa1a5cccb415a094404b88fbfd57"
server_key_iv: "365f7ea30c5886a8b052fdfd5e0b62a2"
server_key_version: "encryptionkey@0"
I20260620 06:15:32.108037 2762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:15:32.110720 2776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:32.111816 2762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 06:15:32.111931 2762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal
uuid: "342dc8d2e3304a08adf63e3d23dd7ac8"
format_stamp: "Formatted at 2026-06-20 06:15:32 on dist-test-slave-zt01"
server_key: "665dfa1a5cccb415a094404b88fbfd57"
server_key_iv: "365f7ea30c5886a8b052fdfd5e0b62a2"
server_key_version: "encryptionkey@0"
I20260620 06:15:32.112035 2762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:32.129205 2762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:32.129743 2762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:32.129909 2762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:32.130396 2762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:15:32.131266 2762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:15:32.131330 2762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:32.131388 2762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:15:32.131434 2762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:32.139707 2762 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.131:45087
I20260620 06:15:32.139816 2889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.131:45087 every 8 connection(s)
I20260620 06:15:32.140522 2762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/data/info.pb
I20260620 06:15:32.146968 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2762
I20260620 06:15:32.147202 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal/instance
I20260620 06:15:32.147471 15174 external_mini_cluster.cc:1428] Setting key 4c77d03076e69e3f8abe6a61a2d1d77d
I20260620 06:15:32.149433 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.142194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.131:55859 (local address 127.14.209.190:43295)
0620 06:15:32.142332 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.142337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.143395 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 06:15:32.144747 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.144750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.144809 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.144883 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:32.145646 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.146161 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.147433 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.147719 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.147828 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.148976 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:32.148980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:32.148982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:32.148992 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:32.149011 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:32.149018 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:32.149078 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.149248 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.149296 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:15:32.150166 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.142556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:43295 (local address 127.14.209.131:55859)
0620 06:15:32.143246 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.143294 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:32.144542 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:32.145023 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:32.145029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:32.145101 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:32.145528 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.145540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.146280 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.146283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.147311 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.147322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.147862 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.147866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.148033 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.148716 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:15:32.148720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:32.148768 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:15:32.148833 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:32.149097 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:32.149105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:32.149110 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:15:32.149228 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 06:15:32.149482 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260620 06:15:32.151130 2890 heartbeater.cc:347] Connected to a master server at 127.14.209.190:43295
I20260620 06:15:32.151355 2890 heartbeater.cc:464] Registering TS with master...
I20260620 06:15:32.151800 2890 heartbeater.cc:511] Master 127.14.209.190:43295 requested a full tablet report, sending...
I20260620 06:15:32.152714 2440 ts_manager.cc:194] Registered new tserver with Master: 342dc8d2e3304a08adf63e3d23dd7ac8 (127.14.209.131:45087)
I20260620 06:15:32.153255 2440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.209.131:55859
I20260620 06:15:32.161162 15174 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:15:32.166484 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.162982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.14.209.190:43295)
0620 06:15:32.163097 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.163101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.163365 (+ 264us) server_negotiation.cc:407] Connection header received
0620 06:15:32.163448 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.163450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.163483 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.163546 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:32.163770 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.164251 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.165044 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.165262 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.165383 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.166063 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:32.166066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:32.166069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:32.166081 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:32.166099 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:32.166113 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:32.166190 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.166299 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.166350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:15:32.168772 2440 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:15:32.171591 2440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:15:32.185277 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.180077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:44349 (local address 127.0.0.1:39680)
0620 06:15:32.180465 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.180477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:32.180561 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:32.181250 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:32.181252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:32.181269 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:32.181348 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.181358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.182290 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.182295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.183740 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.183749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.184784 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.184788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.184906 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.184931 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:15:32.184984 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 06:15:32.185062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":72,"thread_start_us":85,"threads_started":1}
I20260620 06:15:32.185395 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.179791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.14.209.130:46009)
0620 06:15:32.180455 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.180466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.180491 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:15:32.180571 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.180576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.180767 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.181262 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:32.181406 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.182464 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.183856 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.184388 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.184522 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.184694 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.185137 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.185238 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":146,"threads_started":1}
I20260620 06:15:32.185335 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.179357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45087 (local address 127.0.0.1:42768)
0620 06:15:32.179680 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.179729 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:32.179843 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:32.181045 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:32.181047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:32.181063 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:32.181143 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.181151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.182547 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.182550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.183761 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.183767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.184802 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.184805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.185036 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.185060 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:15:32.185139 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:15:32.185203 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":186,"thread_start_us":82,"threads_started":1}
I20260620 06:15:32.185683 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.180010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.14.209.131:45087)
0620 06:15:32.180516 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.180530 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.180547 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:15:32.180601 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.180605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.180766 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.180919 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:32.181261 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.182418 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.183883 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.184695 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.184806 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.185257 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.185326 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.185442 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":89,"threads_started":1}
I20260620 06:15:32.185053 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.179393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46009 (local address 127.0.0.1:56728)
0620 06:15:32.180192 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.180206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:32.180289 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:32.180943 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:32.180949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:32.180970 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:32.181096 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.181102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.182613 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.182616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.183744 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.183753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.184511 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.184514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.184615 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.184649 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 06:15:32.184779 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 06:15:32.184902 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":44,"thread_start_us":83,"threads_started":1}
I20260620 06:15:32.186364 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.180239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.14.209.129:44349)
0620 06:15:32.180820 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.180833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.180858 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:15:32.180933 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.180939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.181114 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.181257 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:32.181474 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.182185 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.183883 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.184643 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.185631 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.186105 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.186148 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.186222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":81,"threads_started":1}
I20260620 06:15:32.188782 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 893cd9db483b4c2f9042d981d2edabbb (DEFAULT_TABLE table=test-table [id=c9e6fe193cd74dc580859ede32469c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:32.188931 2824 tablet_service.cc:1467] Processing CreateTablet for tablet 893cd9db483b4c2f9042d981d2edabbb (DEFAULT_TABLE table=test-table [id=c9e6fe193cd74dc580859ede32469c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:32.189081 2693 tablet_service.cc:1467] Processing CreateTablet for tablet 893cd9db483b4c2f9042d981d2edabbb (DEFAULT_TABLE table=test-table [id=c9e6fe193cd74dc580859ede32469c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:15:32.189749 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893cd9db483b4c2f9042d981d2edabbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:32.189872 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893cd9db483b4c2f9042d981d2edabbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:32.189872 2824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893cd9db483b4c2f9042d981d2edabbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:32.195955 2907 tablet_bootstrap.cc:492] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: Bootstrap starting.
I20260620 06:15:32.197413 2908 tablet_bootstrap.cc:492] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Bootstrap starting.
I20260620 06:15:32.197953 2907 tablet_bootstrap.cc:654] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:32.198617 2907 log.cc:824] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:32.198945 2908 tablet_bootstrap.cc:654] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:32.199290 2909 tablet_bootstrap.cc:492] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Bootstrap starting.
I20260620 06:15:32.199702 2908 log.cc:824] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:32.200989 2907 tablet_bootstrap.cc:492] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: No bootstrap required, opened a new log
I20260620 06:15:32.201189 2907 ts_tablet_manager.cc:1397] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:15:32.201371 2909 tablet_bootstrap.cc:654] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:32.201503 2908 tablet_bootstrap.cc:492] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: No bootstrap required, opened a new log
I20260620 06:15:32.201683 2908 ts_tablet_manager.cc:1397] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 06:15:32.202191 2909 log.cc:824] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:32.204023 2909 tablet_bootstrap.cc:492] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: No bootstrap required, opened a new log
I20260620 06:15:32.204190 2909 ts_tablet_manager.cc:1397] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:15:32.204413 2907 raft_consensus.cc:348] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.204627 2907 raft_consensus.cc:374] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:32.204780 2907 raft_consensus.cc:729] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129faaecf6ce4172b17eba3852fa040b, State: Initialized, Role: FOLLOWER
I20260620 06:15:32.205348 2907 consensus_queue.cc:260] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.206302 2907 ts_tablet_manager.cc:1428] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:15:32.206225 2908 raft_consensus.cc:348] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.206477 2908 raft_consensus.cc:374] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:32.206538 2908 raft_consensus.cc:729] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a19d5523eca466c8459d148191b7377, State: Initialized, Role: FOLLOWER
I20260620 06:15:32.206655 2628 heartbeater.cc:503] Master 127.14.209.190:43295 was elected leader, sending a full tablet report...
I20260620 06:15:32.206923 2908 consensus_queue.cc:260] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.207808 2908 ts_tablet_manager.cc:1428] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:15:32.208127 2759 heartbeater.cc:503] Master 127.14.209.190:43295 was elected leader, sending a full tablet report...
I20260620 06:15:32.208410 2909 raft_consensus.cc:348] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.208599 2909 raft_consensus.cc:374] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:32.208655 2909 raft_consensus.cc:729] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342dc8d2e3304a08adf63e3d23dd7ac8, State: Initialized, Role: FOLLOWER
I20260620 06:15:32.209048 2909 consensus_queue.cc:260] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.209923 2909 ts_tablet_manager.cc:1428] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:15:32.210242 2890 heartbeater.cc:503] Master 127.14.209.190:43295 was elected leader, sending a full tablet report...
W20260620 06:15:32.212481 2760 tablet.cc:2369] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:15:32.294312 2629 tablet.cc:2369] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:32.353807 2913 raft_consensus.cc:482] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:15:32.354044 2913 raft_consensus.cc:504] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.355165 2913 leader_election.cc:288] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 342dc8d2e3304a08adf63e3d23dd7ac8 (127.14.209.131:45087), 3a19d5523eca466c8459d148191b7377 (127.14.209.130:46009)
I20260620 06:15:32.358876 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.355555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:43095 (local address 127.14.209.130:46009)
0620 06:15:32.355694 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.355700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.355870 (+ 170us) server_negotiation.cc:407] Connection header received
0620 06:15:32.356021 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.356024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.356073 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.356159 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:32.356404 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.356913 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.357946 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.358354 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.358480 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.358531 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.358694 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.358758 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:15:32.358896 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.355422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46009 (local address 127.14.209.129:43095)
0620 06:15:32.355786 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.355798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:32.355916 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:32.356170 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:32.356173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:32.356188 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:32.356306 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.356311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.357020 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.357023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.357830 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.357839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.358481 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.358485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.358570 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.358591 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:15:32.358679 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:15:32.358745 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":98,"threads_started":1}
I20260620 06:15:32.359035 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.355524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.209.129:34395 (local address 127.14.209.131:45087)
0620 06:15:32.355669 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.355674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.356001 (+ 327us) server_negotiation.cc:407] Connection header received
0620 06:15:32.356143 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.356146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.356199 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.356283 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:15:32.356566 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.357067 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.358061 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.358480 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.358607 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.358653 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.358823 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.358891 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:15:32.359414 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.355422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.131:45087 (local address 127.14.209.129:34395)
0620 06:15:32.355898 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.355910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:32.356033 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:32.356294 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:32.356297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:32.356312 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:15:32.356442 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.356448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.357177 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.357182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.357937 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:32.357945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.358600 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.358604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:32.358699 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.358725 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:15:32.358808 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:15:32.358875 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":40,"thread_start_us":97,"threads_started":1}
I20260620 06:15:32.359470 2713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893cd9db483b4c2f9042d981d2edabbb" candidate_uuid: "129faaecf6ce4172b17eba3852fa040b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a19d5523eca466c8459d148191b7377" is_pre_election: true
I20260620 06:15:32.359997 2713 raft_consensus.cc:2454] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129faaecf6ce4172b17eba3852fa040b in term 0.
I20260620 06:15:32.360211 2844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893cd9db483b4c2f9042d981d2edabbb" candidate_uuid: "129faaecf6ce4172b17eba3852fa040b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" is_pre_election: true
I20260620 06:15:32.360504 2844 raft_consensus.cc:2454] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129faaecf6ce4172b17eba3852fa040b in term 0.
I20260620 06:15:32.360555 2515 leader_election.cc:302] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129faaecf6ce4172b17eba3852fa040b, 3a19d5523eca466c8459d148191b7377; no voters:
I20260620 06:15:32.360836 2913 raft_consensus.cc:2781] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:15:32.360939 2913 raft_consensus.cc:482] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:15:32.361001 2913 raft_consensus.cc:3037] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:32.362237 2913 raft_consensus.cc:504] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.362565 2913 leader_election.cc:288] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [CANDIDATE]: Term 1 election: Requested vote from peers 342dc8d2e3304a08adf63e3d23dd7ac8 (127.14.209.131:45087), 3a19d5523eca466c8459d148191b7377 (127.14.209.130:46009)
I20260620 06:15:32.362923 2713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893cd9db483b4c2f9042d981d2edabbb" candidate_uuid: "129faaecf6ce4172b17eba3852fa040b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a19d5523eca466c8459d148191b7377"
I20260620 06:15:32.363021 2844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893cd9db483b4c2f9042d981d2edabbb" candidate_uuid: "129faaecf6ce4172b17eba3852fa040b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8"
I20260620 06:15:32.363103 2713 raft_consensus.cc:3037] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:32.363210 2844 raft_consensus.cc:3037] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:32.364114 2713 raft_consensus.cc:2454] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129faaecf6ce4172b17eba3852fa040b in term 1.
I20260620 06:15:32.364167 2844 raft_consensus.cc:2454] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129faaecf6ce4172b17eba3852fa040b in term 1.
I20260620 06:15:32.364420 2515 leader_election.cc:302] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129faaecf6ce4172b17eba3852fa040b, 3a19d5523eca466c8459d148191b7377; no voters:
I20260620 06:15:32.364573 2913 raft_consensus.cc:2781] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:32.364789 2913 raft_consensus.cc:686] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 1 LEADER]: Becoming Leader. State: Replica: 129faaecf6ce4172b17eba3852fa040b, State: Running, Role: LEADER
I20260620 06:15:32.365087 2913 consensus_queue.cc:237] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } }
I20260620 06:15:32.368124 2438 catalog_manager.cc:5495] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b reported cstate change: term changed from 0 to 1, leader changed from <none> to 129faaecf6ce4172b17eba3852fa040b (127.14.209.129). New cstate: current_term: 1 leader_uuid: "129faaecf6ce4172b17eba3852fa040b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "129faaecf6ce4172b17eba3852fa040b" member_type: VOTER last_known_addr { host: "127.14.209.129" port: 44349 } health_report { overall_health: HEALTHY } } }
W20260620 06:15:32.391388 2891 tablet.cc:2369] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:32.421655 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.418185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.14.209.129:44349)
0620 06:15:32.418349 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:32.418353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:32.418365 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:15:32.418437 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:32.418439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:32.418492 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:32.418577 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:15:32.418787 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.419324 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.420176 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:32.420387 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:32.420501 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:32.420852 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:15:32.421257 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:15:32.421356 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:32.421466 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 06:15:32.421514 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 06:15:32.426983 2713 raft_consensus.cc:1261] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 1 FOLLOWER]: Refusing update from remote peer 129faaecf6ce4172b17eba3852fa040b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:32.426983 2844 raft_consensus.cc:1261] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 1 FOLLOWER]: Refusing update from remote peer 129faaecf6ce4172b17eba3852fa040b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:32.427733 2918 consensus_queue.cc:1035] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a19d5523eca466c8459d148191b7377" member_type: VOTER last_known_addr { host: "127.14.209.130" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:32.427904 2913 consensus_queue.cc:1035] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [LEADER]: Connected to new peer: Peer: permanent_uuid: "342dc8d2e3304a08adf63e3d23dd7ac8" member_type: VOTER last_known_addr { host: "127.14.209.131" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:32.436563 2923 mvcc.cc:205] Tried to move back new op lower bound from 7298810398414544896 to 7298810398169903104. Current Snapshot: MvccSnapshot[applied={T|T < 7298810398414544896 or (T in {7298810398414544896})}]
I20260620 06:15:32.442312 2438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:32.442464 2438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:15:32.444706 2438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c247656513d2412a8ff88e09149464d1: Sending DeleteTablet for 3 replicas of tablet 893cd9db483b4c2f9042d981d2edabbb
I20260620 06:15:32.445432 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 893cd9db483b4c2f9042d981d2edabbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:32 UTC) from {username='slave'} at 127.0.0.1:39680
I20260620 06:15:32.445418 2693 tablet_service.cc:1514] Processing DeleteTablet for tablet 893cd9db483b4c2f9042d981d2edabbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:32 UTC) from {username='slave'} at 127.0.0.1:56728
I20260620 06:15:32.445861 2932 tablet_replica.cc:330] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b: stopping tablet replica
I20260620 06:15:32.445848 2824 tablet_service.cc:1514] Processing DeleteTablet for tablet 893cd9db483b4c2f9042d981d2edabbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:15:32 UTC) from {username='slave'} at 127.0.0.1:42768
I20260620 06:15:32.446138 2932 raft_consensus.cc:2229] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:15:32.446310 2933 tablet_replica.cc:330] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: stopping tablet replica
I20260620 06:15:32.446435 2932 raft_consensus.cc:2258] T 893cd9db483b4c2f9042d981d2edabbb P 129faaecf6ce4172b17eba3852fa040b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:32.446863 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2500
I20260620 06:15:32.446652 2933 raft_consensus.cc:2229] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:32.447646 2933 raft_consensus.cc:2258] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:32.448565 2931 tablet_replica.cc:330] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: stopping tablet replica
I20260620 06:15:32.448802 2931 raft_consensus.cc:2229] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:15:32.449039 2931 raft_consensus.cc:2258] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:15:32.449237 2933 ts_tablet_manager.cc:1905] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:32.451507 2933 ts_tablet_manager.cc:1918] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:15:32.451604 2933 log.cc:1196] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Deleting WAL directory at /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935885709390-15174-0/minicluster-data/ts-2/wal/wals/893cd9db483b4c2f9042d981d2edabbb
I20260620 06:15:32.451905 2933 ts_tablet_manager.cc:1939] T 893cd9db483b4c2f9042d981d2edabbb P 342dc8d2e3304a08adf63e3d23dd7ac8: Deleting consensus metadata
I20260620 06:15:32.452147 2931 ts_tablet_manager.cc:1905] T 893cd9db483b4c2f9042d981d2edabbb P 3a19d5523eca466c8459d148191b7377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:15:32.453184 2425 catalog_manager.cc:4841] TS 342dc8d2e3304a08adf63e3d23dd7ac8 (127.14.209.131:45087): tablet 893cd9db483b4c2f9042d981d2edabbb (table test-table [id=c9e6fe193cd74dc580859ede32469c65]) successfully deleted
I20260620 06:15:32.453966 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2631
W20260620 06:15:32.454026 2425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.209.129:44349 (error 108)
I20260620 06:15:32.456158 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.455827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.129:44349 (local address 127.0.0.1:39708)
0620 06:15:32.455981 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.456064 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.129:44349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:15:32.456301 2425 catalog_manager.cc:4584] TS 129faaecf6ce4172b17eba3852fa040b (127.14.209.129:44349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 893cd9db483b4c2f9042d981d2edabbb: Network error: Client connection negotiation failed: client connection to 127.14.209.129:44349: connect: Connection refused (error 111)
I20260620 06:15:32.459899 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2762
I20260620 06:15:32.461555 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:32.460136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.130:46009 (local address 127.0.0.1:56738)
0620 06:15:32.461408 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:32.461472 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.130:46009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1181}
I20260620 06:15:32.465356 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2409
2026-06-20T06:15:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1121 ms)
[----------] 6 tests from EncryptionPolicyTest (10642 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:32 dist-test-slave-zt01 krb5kdc[2937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:32 dist-test-slave-zt01 krb5kdc[2937](info): set up 2 sockets
Jun 20 06:15:32 dist-test-slave-zt01 krb5kdc[2937](info): commencing operation
krb5kdc: starting...
W20260620 06:15:33.499143 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:33 dist-test-slave-zt01 krb5kdc[2937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:33.614284 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:33.718525 2950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:33.718812 2950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:33.718899 2950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:33.722759 2950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:33.722847 2950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:33.722891 2950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:33.722950 2950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:33.722990 2950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:33.728399 2950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:33.729303 2950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:33.730203 2950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:33.737969 2956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:33.738035 2958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:33.737960 2955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:33.738270 2950 server_base.cc:1034] running on GCE node
I20260620 06:15:33.738814 2950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:33.739871 2950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:33.741034 2950 hybrid_clock.cc:648] HybridClock initialized: now 1781936133741017 us; error 32 us; skew 500 ppm
Jun 20 06:15:33 dist-test-slave-zt01 krb5kdc[2937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:33.748487 2950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:33.749325 2950 webserver.cc:460] Webserver started at http://127.0.0.1:45319/ using document root <none> and password file <none>
I20260620 06:15:33.749836 2950 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:33.749922 2950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:33.750129 2950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:33.752043 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "8e343254407941eca88503b683606564"
format_stamp: "Formatted at 2026-06-20 06:15:33 on dist-test-slave-zt01"
server_key: "e3a06c50401200ec44badf1d2cb028e7"
server_key_iv: "1edd83b4ab9c76685c87d9ebac36b78d"
server_key_version: "encryptionkey@0"
I20260620 06:15:33.752558 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "8e343254407941eca88503b683606564"
format_stamp: "Formatted at 2026-06-20 06:15:33 on dist-test-slave-zt01"
server_key: "e3a06c50401200ec44badf1d2cb028e7"
server_key_iv: "1edd83b4ab9c76685c87d9ebac36b78d"
server_key_version: "encryptionkey@0"
I20260620 06:15:33.756063 2950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:15:33.758158 2965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:33.759370 2950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:33.759495 2950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "8e343254407941eca88503b683606564"
format_stamp: "Formatted at 2026-06-20 06:15:33 on dist-test-slave-zt01"
server_key: "e3a06c50401200ec44badf1d2cb028e7"
server_key_iv: "1edd83b4ab9c76685c87d9ebac36b78d"
server_key_version: "encryptionkey@0"
I20260620 06:15:33.759595 2950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:33.774866 2950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:33.777913 2950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:33.778098 2950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:33.784452 2950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40639
I20260620 06:15:33.784468 3017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40639 every 8 connection(s)
I20260620 06:15:33.785295 2950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:33.787843 3018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:33.789844 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 2950
I20260620 06:15:33.790000 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:33.790256 15174 external_mini_cluster.cc:1428] Setting key c98a467a6a382ac66e90f537069a02cd
I20260620 06:15:33.793157 3018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564: Bootstrap starting.
I20260620 06:15:33.795145 3018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:33.795843 3018 log.cc:824] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:33.797482 3018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564: No bootstrap required, opened a new log
Jun 20 06:15:33 dist-test-slave-zt01 krb5kdc[2937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:15:33.800285 3018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e343254407941eca88503b683606564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 06:15:33.800495 3018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:33.800555 3018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e343254407941eca88503b683606564, State: Initialized, Role: FOLLOWER
I20260620 06:15:33.801011 3018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e343254407941eca88503b683606564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 06:15:33.801133 3018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:33.801208 3018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:33.801322 3018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:33.802091 3018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e343254407941eca88503b683606564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 06:15:33.802412 3018 leader_election.cc:302] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e343254407941eca88503b683606564; no voters:
I20260620 06:15:33.802682 3018 leader_election.cc:288] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:33.802779 3023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:33.802986 3023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [term 1 LEADER]: Becoming Leader. State: Replica: 8e343254407941eca88503b683606564, State: Running, Role: LEADER
I20260620 06:15:33.803375 3023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e343254407941eca88503b683606564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260620 06:15:33.803900 3018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:33.804845 3024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e343254407941eca88503b683606564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e343254407941eca88503b683606564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260620 06:15:33.804965 3024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:33.804983 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:33.791517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60497 (local address 127.0.0.1:40639)
0620 06:15:33.791910 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:33.791924 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:33.791956 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:15:33.792556 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:33.792570 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:33.792859 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:33.793164 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:33.793955 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:33.794889 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:33.795849 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:33.796498 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:33.796642 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:33.799123 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:33.799129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:33.799150 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:33.799182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:33.801486 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:33.801960 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:33.801966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:33.801971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:33.802081 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:33.802410 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:33.802414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:33.802415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:33.802955 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:33.803193 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:33.803862 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 06:15:33.804074 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260620 06:15:33.805285 3025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e343254407941eca88503b683606564. Latest consensus state: current_term: 1 leader_uuid: "8e343254407941eca88503b683606564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e343254407941eca88503b683606564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260620 06:15:33.805365 3025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:33.807966 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 06:15:33.809317 3038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:33.809394 3038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:33.809459 3032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:33.810320 3032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:33.815241 3032 catalog_manager.cc:1347] Generated new cluster ID: e7fcc63cb70d42dcb4283c9d5c695480
I20260620 06:15:33.815303 3032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:33.821873 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:33.809883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58195 (local address 127.0.0.1:40639)
0620 06:15:33.810005 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:33.810010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:33.813801 (+ 3791us) server_negotiation.cc:407] Connection header received
0620 06:15:33.813945 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:33.813953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:33.813990 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:33.814070 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:33.814679 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:33.815371 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:33.816496 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:33.816834 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:33.816926 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:33.818509 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:33.818513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:33.818515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:33.818538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:33.820127 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:33.820646 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:33.820649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:33.820651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:33.820727 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:33.821065 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:33.821068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:33.821070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:33.821243 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:33.821364 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:33.821643 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 06:15:33.821728 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:15:33.831658 3032 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:33.832767 3032 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:33.840587 3032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e343254407941eca88503b683606564: Generated new TSK 0
I20260620 06:15:33.841238 3032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:33.844512 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 2950
W20260620 06:15:33.849674 2935 connection.cc:535] client connection to 127.0.0.1:40639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40639): Connection reset by peer (error 104)
2026-06-20T06:15:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:33 dist-test-slave-zt01 krb5kdc[3049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:33 dist-test-slave-zt01 krb5kdc[3049](info): set up 2 sockets
Jun 20 06:15:33 dist-test-slave-zt01 krb5kdc[3049](info): commencing operation
krb5kdc: starting...
W20260620 06:15:34.887244 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:34 dist-test-slave-zt01 krb5kdc[3049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:35.002300 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:35.112057 3062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:35.112282 3062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:35.112324 3062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:35.118448 3062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:35.118523 3062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:35.118558 3062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:35.118595 3062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:35.118633 3062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:35.126016 3062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:35.127005 3062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:35.127974 3062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:35.135911 3068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:35.135996 3070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:35.135927 3067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:35.136487 3062 server_base.cc:1034] running on GCE node
I20260620 06:15:35.136987 3062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:35.137979 3062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:35.139132 3062 hybrid_clock.cc:648] HybridClock initialized: now 1781936135139061 us; error 74 us; skew 500 ppm
Jun 20 06:15:35 dist-test-slave-zt01 krb5kdc[3049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:35.141693 3062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:35.142484 3062 webserver.cc:460] Webserver started at http://127.0.0.1:45575/ using document root <none> and password file <none>
I20260620 06:15:35.143026 3062 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:35.143169 3062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:35.143397 3062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:35.145290 3062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "8209e9eeeb0d45c4a11565c040e77755"
format_stamp: "Formatted at 2026-06-20 06:15:35 on dist-test-slave-zt01"
server_key: "a2e5a95c70311d9d1d9c51facace2043"
server_key_iv: "d8ed7b1ce905534d71204e18ab20e4d4"
server_key_version: "encryptionkey@0"
I20260620 06:15:35.145829 3062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "8209e9eeeb0d45c4a11565c040e77755"
format_stamp: "Formatted at 2026-06-20 06:15:35 on dist-test-slave-zt01"
server_key: "a2e5a95c70311d9d1d9c51facace2043"
server_key_iv: "d8ed7b1ce905534d71204e18ab20e4d4"
server_key_version: "encryptionkey@0"
I20260620 06:15:35.149329 3062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:35.151595 3077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:35.152709 3062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:35.152848 3062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "8209e9eeeb0d45c4a11565c040e77755"
format_stamp: "Formatted at 2026-06-20 06:15:35 on dist-test-slave-zt01"
server_key: "a2e5a95c70311d9d1d9c51facace2043"
server_key_iv: "d8ed7b1ce905534d71204e18ab20e4d4"
server_key_version: "encryptionkey@0"
I20260620 06:15:35.152953 3062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:35.168859 3062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:35.171872 3062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:35.172055 3062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:35.179183 3062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41621
I20260620 06:15:35.179194 3129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41621 every 8 connection(s)
I20260620 06:15:35.180064 3062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:35.182572 3130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:35.187559 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3062
I20260620 06:15:35.187650 3130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755: Bootstrap starting.
I20260620 06:15:35.187721 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:35.188000 15174 external_mini_cluster.cc:1428] Setting key 88cf83765a1b37b737b67bd0e0e40a69
I20260620 06:15:35.190135 3130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:35.191046 3130 log.cc:824] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:35.192994 3130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755: No bootstrap required, opened a new log
I20260620 06:15:35.196331 3130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8209e9eeeb0d45c4a11565c040e77755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260620 06:15:35.196563 3130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:35.196656 3130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8209e9eeeb0d45c4a11565c040e77755, State: Initialized, Role: FOLLOWER
I20260620 06:15:35.197060 3130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8209e9eeeb0d45c4a11565c040e77755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260620 06:15:35.197168 3130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:35.197248 3130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:35.197263 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:35.189278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34053 (local address 127.0.0.1:41621)
0620 06:15:35.189677 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:35.189691 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:35.189723 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:15:35.190695 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:35.190717 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:35.191225 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:35.191577 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:35.191986 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:35.193025 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:35.193902 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:35.194368 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:35.194545 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:35.195433 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:35.195442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:35.195460 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:35.195846 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:35.195897 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:35.195919 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:35.196057 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:35.196222 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 06:15:35.196354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
I20260620 06:15:35.197348 3130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:35.198192 3130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8209e9eeeb0d45c4a11565c040e77755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260620 06:15:35.198513 3130 leader_election.cc:302] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8209e9eeeb0d45c4a11565c040e77755; no voters:
I20260620 06:15:35.198817 3130 leader_election.cc:288] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:35.198930 3135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:35.199239 3135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [term 1 LEADER]: Becoming Leader. State: Replica: 8209e9eeeb0d45c4a11565c040e77755, State: Running, Role: LEADER
I20260620 06:15:35.199594 3135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8209e9eeeb0d45c4a11565c040e77755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } }
I20260620 06:15:35.200022 3130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:35.200147 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:35.201246 3137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8209e9eeeb0d45c4a11565c040e77755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8209e9eeeb0d45c4a11565c040e77755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } } }
I20260620 06:15:35.201367 3137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:35.201731 3136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8209e9eeeb0d45c4a11565c040e77755. Latest consensus state: current_term: 1 leader_uuid: "8209e9eeeb0d45c4a11565c040e77755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8209e9eeeb0d45c4a11565c040e77755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41621 } } }
I20260620 06:15:35.201882 3136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:35.202641 3147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:35.205428 3147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:35.207288 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:35.202716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54595 (local address 127.0.0.1:41621)
0620 06:15:35.202825 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:35.202829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:35.203222 (+ 393us) server_negotiation.cc:407] Connection header received
0620 06:15:35.203326 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:35.203329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:35.203375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:35.203453 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:35.203696 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:35.204422 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:35.205524 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:35.205857 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:35.205960 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:35.206792 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:35.206798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:35.206802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:35.206814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:35.206840 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:35.206855 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:35.206945 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:35.207142 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 06:15:35.207173 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:15:35.210773 3147 catalog_manager.cc:1347] Generated new cluster ID: 9db84bf498764c369cc5f5a2e6a575f9
I20260620 06:15:35.210853 3147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:35.231637 3147 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:35.232699 3147 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:35.238626 3147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8209e9eeeb0d45c4a11565c040e77755: Generated new TSK 0
I20260620 06:15:35.239289 3147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:35.247565 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3062
W20260620 06:15:35.252972 3047 connection.cc:535] client connection to 127.0.0.1:41621 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41621): Connection reset by peer (error 104)
2026-06-20T06:15:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:35 dist-test-slave-zt01 krb5kdc[3161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:35 dist-test-slave-zt01 krb5kdc[3161](info): set up 2 sockets
Jun 20 06:15:35 dist-test-slave-zt01 krb5kdc[3161](info): commencing operation
krb5kdc: starting...
W20260620 06:15:36.287096 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:36 dist-test-slave-zt01 krb5kdc[3161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:36.406240 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:36.512125 3174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:36.512362 3174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:36.512403 3174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:36.516374 3174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:36.516426 3174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:36.516446 3174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:36.516465 3174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:36.516685 3174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:36.524156 3174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:36.525497 3174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:36.526537 3174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:36.535171 3182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:36.535176 3180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:36.535743 3179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:36.535925 3174 server_base.cc:1034] running on GCE node
I20260620 06:15:36.536513 3174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:36.537621 3174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:36.538829 3174 hybrid_clock.cc:648] HybridClock initialized: now 1781936136538813 us; error 31 us; skew 500 ppm
Jun 20 06:15:36 dist-test-slave-zt01 krb5kdc[3161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:36.541524 3174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:36.542407 3174 webserver.cc:460] Webserver started at http://127.0.0.1:44297/ using document root <none> and password file <none>
I20260620 06:15:36.542945 3174 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:36.543035 3174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:36.543259 3174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:36.545181 3174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "3be97b7794e641f78ba4397058281aa0"
format_stamp: "Formatted at 2026-06-20 06:15:36 on dist-test-slave-zt01"
server_key: "037f274c33c04567abca7f5227d37c49"
server_key_iv: "f079e6ae8f16fd2c412a8fbe42cc8385"
server_key_version: "encryptionkey@0"
I20260620 06:15:36.545717 3174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "3be97b7794e641f78ba4397058281aa0"
format_stamp: "Formatted at 2026-06-20 06:15:36 on dist-test-slave-zt01"
server_key: "037f274c33c04567abca7f5227d37c49"
server_key_iv: "f079e6ae8f16fd2c412a8fbe42cc8385"
server_key_version: "encryptionkey@0"
I20260620 06:15:36.549189 3174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:15:36.551455 3189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:36.552629 3174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:36.552767 3174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "3be97b7794e641f78ba4397058281aa0"
format_stamp: "Formatted at 2026-06-20 06:15:36 on dist-test-slave-zt01"
server_key: "037f274c33c04567abca7f5227d37c49"
server_key_iv: "f079e6ae8f16fd2c412a8fbe42cc8385"
server_key_version: "encryptionkey@0"
I20260620 06:15:36.552875 3174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:36.558182 3174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:36.561117 3174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:36.561311 3174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:36.568306 3174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44593
I20260620 06:15:36.568321 3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44593 every 8 connection(s)
I20260620 06:15:36.569195 3174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:36.571187 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3174
I20260620 06:15:36.571341 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:36.571602 15174 external_mini_cluster.cc:1428] Setting key 29550d6619ea6f4d81e055780df95663
I20260620 06:15:36.572422 3242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:36.578401 3242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0: Bootstrap starting.
I20260620 06:15:36.578852 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:36.572841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54435 (local address 127.0.0.1:44593)
0620 06:15:36.573268 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:36.573287 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:36.575380 (+ 2093us) server_negotiation.cc:407] Connection header received
0620 06:15:36.576245 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:36.576266 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:36.576603 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:36.576935 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:36.577598 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:36.577604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:36.577617 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:36.577628 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:36.577685 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:36.577711 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:36.577843 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:36.577914 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 06:15:36.578044 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
I20260620 06:15:36.580766 3242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:36.581573 3242 log.cc:824] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:36.581617 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:36.583467 3242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0: No bootstrap required, opened a new log
I20260620 06:15:36.585134 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:36.583666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49365 (local address 127.0.0.1:44593)
0620 06:15:36.583780 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:36.583786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:36.583931 (+ 145us) server_negotiation.cc:407] Connection header received
0620 06:15:36.584008 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:36.584011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:36.584042 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:36.584131 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:36.584811 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:36.584815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:36.584817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:36.584828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:36.584853 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:36.584867 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:36.584961 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:36.584995 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 06:15:36.585027 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:15:36.587494 3242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be97b7794e641f78ba4397058281aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260620 06:15:36.587759 3242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:36.587846 3242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be97b7794e641f78ba4397058281aa0, State: Initialized, Role: FOLLOWER
I20260620 06:15:36.588251 3242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be97b7794e641f78ba4397058281aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260620 06:15:36.588392 3242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:36.588459 3242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:36.588551 3242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:36.589351 3242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be97b7794e641f78ba4397058281aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260620 06:15:36.589669 3242 leader_election.cc:302] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be97b7794e641f78ba4397058281aa0; no voters:
I20260620 06:15:36.589941 3242 leader_election.cc:288] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:36.590076 3252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:36.590301 3252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [term 1 LEADER]: Becoming Leader. State: Replica: 3be97b7794e641f78ba4397058281aa0, State: Running, Role: LEADER
I20260620 06:15:36.590612 3252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be97b7794e641f78ba4397058281aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260620 06:15:36.591022 3242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:36.592099 3254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be97b7794e641f78ba4397058281aa0. Latest consensus state: current_term: 1 leader_uuid: "3be97b7794e641f78ba4397058281aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be97b7794e641f78ba4397058281aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } } }
I20260620 06:15:36.592226 3254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:36.592612 3253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be97b7794e641f78ba4397058281aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be97b7794e641f78ba4397058281aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } } }
I20260620 06:15:36.592728 3253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0 [sys.catalog]: This master's current role is: LEADER
W20260620 06:15:36.595201 3267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:36.595279 3267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:36.595352 3261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:36.596021 3261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:36.601459 3261 catalog_manager.cc:1347] Generated new cluster ID: 1cb6bebe174043e39f7eccf6be6626a3
I20260620 06:15:36.601519 3261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:36.607334 3261 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:36.607596 3261 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:36.612664 3261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be97b7794e641f78ba4397058281aa0: Generated new TSK 0
I20260620 06:15:36.613297 3261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:36.625159 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3174
2026-06-20T06:15:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:36 dist-test-slave-zt01 krb5kdc[3273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:36 dist-test-slave-zt01 krb5kdc[3273](info): set up 2 sockets
Jun 20 06:15:36 dist-test-slave-zt01 krb5kdc[3273](info): commencing operation
krb5kdc: starting...
W20260620 06:15:37.659149 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:37 dist-test-slave-zt01 krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:37.773391 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:37.877861 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:37.878079 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:37.878119 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:37.882112 3286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:37.882164 3286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:37.882182 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:37.882200 3286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:37.882241 3286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:37.887295 3286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:37.888234 3286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:37.889140 3286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:37.896931 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:37.897017 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:37.896931 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:37.897147 3286 server_base.cc:1034] running on GCE node
I20260620 06:15:37.897768 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:37.898886 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:37.900039 3286 hybrid_clock.cc:648] HybridClock initialized: now 1781936137900014 us; error 39 us; skew 500 ppm
Jun 20 06:15:37 dist-test-slave-zt01 krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:37.902707 3286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:37.903609 3286 webserver.cc:460] Webserver started at http://127.0.0.1:45531/ using document root <none> and password file <none>
I20260620 06:15:37.904129 3286 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:37.904215 3286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:37.904433 3286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:37.906286 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "1d0b235c04364330b03abb35cfcd2ddb"
format_stamp: "Formatted at 2026-06-20 06:15:37 on dist-test-slave-zt01"
server_key: "9cfbe28fffaca874144f93d37145abd6"
server_key_iv: "f1ca3b5c7a8b7423080b3ffcfe9b369b"
server_key_version: "encryptionkey@0"
I20260620 06:15:37.906827 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "1d0b235c04364330b03abb35cfcd2ddb"
format_stamp: "Formatted at 2026-06-20 06:15:37 on dist-test-slave-zt01"
server_key: "9cfbe28fffaca874144f93d37145abd6"
server_key_iv: "f1ca3b5c7a8b7423080b3ffcfe9b369b"
server_key_version: "encryptionkey@0"
I20260620 06:15:37.910261 3286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:15:37.912518 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:37.913640 3286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:37.913774 3286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "1d0b235c04364330b03abb35cfcd2ddb"
format_stamp: "Formatted at 2026-06-20 06:15:37 on dist-test-slave-zt01"
server_key: "9cfbe28fffaca874144f93d37145abd6"
server_key_iv: "f1ca3b5c7a8b7423080b3ffcfe9b369b"
server_key_version: "encryptionkey@0"
I20260620 06:15:37.913892 3286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:37.927587 3286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:37.930593 3286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:37.930784 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:37.937506 3286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37665
I20260620 06:15:37.937537 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37665 every 8 connection(s)
I20260620 06:15:37.938370 3286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:37.938675 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3286
I20260620 06:15:37.938835 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:37.939203 15174 external_mini_cluster.cc:1428] Setting key b6d1c8a5d586825e3e65b9f95b6f81fc
I20260620 06:15:37.941517 3354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 06:15:37 dist-test-slave-zt01 krb5kdc[3273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:15:37.948340 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb: Bootstrap starting.
I20260620 06:15:37.950414 3354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:37.951196 3354 log.cc:824] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:37.953163 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb: No bootstrap required, opened a new log
I20260620 06:15:37.953161 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:37.940585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52021 (local address 127.0.0.1:37665)
0620 06:15:37.940960 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:37.940976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:37.941018 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:15:37.941594 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:37.941608 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:37.941967 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:37.942219 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:37.943221 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:37.944143 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:37.945449 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:37.945888 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:37.946001 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:37.948816 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:37.948820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:37.948831 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:37.948855 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:37.950450 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:37.950892 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:37.950897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:37.950901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:37.951003 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:37.951311 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:37.951314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:37.951316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:37.951748 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:37.951959 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:37.952249 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 06:15:37.952515 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260620 06:15:37.955965 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:37.957324 3354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0b235c04364330b03abb35cfcd2ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260620 06:15:37.957556 3354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:37.957638 3354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0b235c04364330b03abb35cfcd2ddb, State: Initialized, Role: FOLLOWER
I20260620 06:15:37.958153 3354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0b235c04364330b03abb35cfcd2ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260620 06:15:37.958335 3354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:37.958425 3354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:37.958536 3354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:37.959594 3354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0b235c04364330b03abb35cfcd2ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260620 06:15:37.959951 3354 leader_election.cc:302] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d0b235c04364330b03abb35cfcd2ddb; no voters:
I20260620 06:15:37.960317 3354 leader_election.cc:288] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:37.960440 3364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:37.960644 3364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [term 1 LEADER]: Becoming Leader. State: Replica: 1d0b235c04364330b03abb35cfcd2ddb, State: Running, Role: LEADER
I20260620 06:15:37.961043 3364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0b235c04364330b03abb35cfcd2ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260620 06:15:37.961521 3354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:37.962177 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d0b235c04364330b03abb35cfcd2ddb. Latest consensus state: current_term: 1 leader_uuid: "1d0b235c04364330b03abb35cfcd2ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0b235c04364330b03abb35cfcd2ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } } }
I20260620 06:15:37.962308 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:37.962746 3370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:37.962725 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d0b235c04364330b03abb35cfcd2ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0b235c04364330b03abb35cfcd2ddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } } }
I20260620 06:15:37.964154 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:37.966310 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:37.957823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40187 (local address 127.0.0.1:37665)
0620 06:15:37.957943 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:37.957947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:37.958116 (+ 169us) server_negotiation.cc:407] Connection header received
0620 06:15:37.958185 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:37.958188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:37.958231 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:37.958321 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:37.959092 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:37.959897 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:37.960970 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:37.961283 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:37.961358 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:37.963007 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:37.963012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:37.963015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:37.963039 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:37.964741 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:37.965186 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:37.965190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:37.965191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:37.965271 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:37.965531 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:37.965534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:37.965536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:37.965690 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:37.965820 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:37.966051 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 06:15:37.966154 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:15:37.966575 3370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:37.971894 3370 catalog_manager.cc:1347] Generated new cluster ID: 707a41ea65ae47939bd6f32e2e330617
I20260620 06:15:37.971951 3370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:37.986266 3370 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:37.987097 3370 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:37.999275 3370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d0b235c04364330b03abb35cfcd2ddb: Generated new TSK 0
I20260620 06:15:37.999882 3370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:38.010623 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3286
2026-06-20T06:15:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:38 dist-test-slave-zt01 krb5kdc[3385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:38 dist-test-slave-zt01 krb5kdc[3385](info): set up 2 sockets
Jun 20 06:15:38 dist-test-slave-zt01 krb5kdc[3385](info): commencing operation
krb5kdc: starting...
W20260620 06:15:39.043234 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:39 dist-test-slave-zt01 krb5kdc[3385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:39.163975 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:39.269729 3398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:39.269948 3398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:39.269986 3398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:39.273834 3398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:39.273883 3398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:39.273903 3398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:39.273919 3398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:39.273934 3398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:39.279208 3398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:39.280067 3398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:39.280929 3398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:39.288883 3404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:39.288853 3406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:39.288877 3403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:39.289163 3398 server_base.cc:1034] running on GCE node
I20260620 06:15:39.289697 3398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:39.290722 3398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:39.291909 3398 hybrid_clock.cc:648] HybridClock initialized: now 1781936139291885 us; error 40 us; skew 500 ppm
Jun 20 06:15:39 dist-test-slave-zt01 krb5kdc[3385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:39.299912 3398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:39.300794 3398 webserver.cc:460] Webserver started at http://127.0.0.1:36371/ using document root <none> and password file <none>
I20260620 06:15:39.301409 3398 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:39.301499 3398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:39.301723 3398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:39.303750 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "9c15984a11e24b2c8d7be4da5959a22a"
format_stamp: "Formatted at 2026-06-20 06:15:39 on dist-test-slave-zt01"
server_key: "f21e69a49da0dd676d9ff85d42ea1689"
server_key_iv: "5757d62656c91b5ee2066efa8d427f1f"
server_key_version: "encryptionkey@0"
I20260620 06:15:39.304500 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "9c15984a11e24b2c8d7be4da5959a22a"
format_stamp: "Formatted at 2026-06-20 06:15:39 on dist-test-slave-zt01"
server_key: "f21e69a49da0dd676d9ff85d42ea1689"
server_key_iv: "5757d62656c91b5ee2066efa8d427f1f"
server_key_version: "encryptionkey@0"
I20260620 06:15:39.308075 3398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:15:39.310333 3413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:39.311671 3398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:39.311829 3398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "9c15984a11e24b2c8d7be4da5959a22a"
format_stamp: "Formatted at 2026-06-20 06:15:39 on dist-test-slave-zt01"
server_key: "f21e69a49da0dd676d9ff85d42ea1689"
server_key_iv: "5757d62656c91b5ee2066efa8d427f1f"
server_key_version: "encryptionkey@0"
I20260620 06:15:39.311939 3398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:39.322584 3398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:39.325232 3398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:39.325423 3398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:39.332535 3398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36879
I20260620 06:15:39.332561 3465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36879 every 8 connection(s)
I20260620 06:15:39.333403 3398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:39.335983 3466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:39.339341 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3398
I20260620 06:15:39.339486 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:39.339799 15174 external_mini_cluster.cc:1428] Setting key d834438eb78af74d47b5d27768c03ca3
I20260620 06:15:39.341311 3466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a: Bootstrap starting.
I20260620 06:15:39.343345 3466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:39.344020 3466 log.cc:824] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:39.345646 3466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a: No bootstrap required, opened a new log
I20260620 06:15:39.348021 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:39.341051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40239 (local address 127.0.0.1:36879)
0620 06:15:39.341502 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:39.341517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:39.341549 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:15:39.342135 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:39.342149 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:39.342454 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:39.342759 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:39.343208 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:39.344263 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:39.345131 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:39.345573 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:39.345736 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:39.346515 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:39.346520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:39.346531 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:39.346880 (+ 349us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:39.346929 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:39.346950 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:39.347084 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:39.347227 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:15:39.347406 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260620 06:15:39.349388 3466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c15984a11e24b2c8d7be4da5959a22a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260620 06:15:39.349651 3466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:39.349714 3466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c15984a11e24b2c8d7be4da5959a22a, State: Initialized, Role: FOLLOWER
I20260620 06:15:39.350209 3466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c15984a11e24b2c8d7be4da5959a22a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260620 06:15:39.350359 3466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:39.350426 3466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:39.350521 3466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:39.350651 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:39.351397 3466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c15984a11e24b2c8d7be4da5959a22a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260620 06:15:39.351706 3466 leader_election.cc:302] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c15984a11e24b2c8d7be4da5959a22a; no voters:
I20260620 06:15:39.352033 3466 leader_election.cc:288] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:39.352114 3475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:39.352337 3475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [term 1 LEADER]: Becoming Leader. State: Replica: 9c15984a11e24b2c8d7be4da5959a22a, State: Running, Role: LEADER
I20260620 06:15:39.352653 3475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c15984a11e24b2c8d7be4da5959a22a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } }
I20260620 06:15:39.353299 3466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:39.353834 3475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c15984a11e24b2c8d7be4da5959a22a. Latest consensus state: current_term: 1 leader_uuid: "9c15984a11e24b2c8d7be4da5959a22a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c15984a11e24b2c8d7be4da5959a22a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } } }
I20260620 06:15:39.354012 3475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:39.354307 3476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c15984a11e24b2c8d7be4da5959a22a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c15984a11e24b2c8d7be4da5959a22a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36879 } } }
I20260620 06:15:39.354449 3476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:39.355321 3484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:39.357223 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:39.352610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33157 (local address 127.0.0.1:36879)
0620 06:15:39.352705 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:39.352709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:39.352918 (+ 209us) server_negotiation.cc:407] Connection header received
0620 06:15:39.353082 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:39.353084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:39.353127 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:39.353210 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:39.353493 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:39.354265 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:39.355475 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:39.355824 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:39.355883 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:39.356754 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:39.356757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:39.356760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:39.356772 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:39.356803 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:39.356813 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:39.356892 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:39.357022 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 06:15:39.357067 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 06:15:39.358693 3484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:39.363912 3484 catalog_manager.cc:1347] Generated new cluster ID: 0a4565ba8f464fd2a7cb880d63e15eff
I20260620 06:15:39.363982 3484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:39.383561 3484 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:39.384478 3484 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:39.392966 3484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c15984a11e24b2c8d7be4da5959a22a: Generated new TSK 0
I20260620 06:15:39.393581 3484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:39.400503 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3398
2026-06-20T06:15:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:39 dist-test-slave-zt01 krb5kdc[3497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:39 dist-test-slave-zt01 krb5kdc[3497](info): set up 2 sockets
Jun 20 06:15:39 dist-test-slave-zt01 krb5kdc[3497](info): commencing operation
krb5kdc: starting...
W20260620 06:15:40.435470 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:40 dist-test-slave-zt01 krb5kdc[3497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:40.553556 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:40.658259 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:40.658470 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:40.658515 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:40.662264 3510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:40.662308 3510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:40.662328 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:40.662343 3510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:40.662358 3510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:40.667171 3510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:40.667944 3510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:40.668748 3510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:40.676433 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:40.676436 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:40.676510 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:40.676590 3510 server_base.cc:1034] running on GCE node
I20260620 06:15:40.677228 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:40.678231 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:40.679425 3510 hybrid_clock.cc:648] HybridClock initialized: now 1781936140679395 us; error 40 us; skew 500 ppm
Jun 20 06:15:40 dist-test-slave-zt01 krb5kdc[3497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:40.681977 3510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:40.682711 3510 webserver.cc:460] Webserver started at http://127.0.0.1:46017/ using document root <none> and password file <none>
I20260620 06:15:40.683246 3510 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:40.683306 3510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:40.683451 3510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:40.685216 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "f13093324e444e9aa27a429e2ca99e4d"
format_stamp: "Formatted at 2026-06-20 06:15:40 on dist-test-slave-zt01"
server_key: "e0c39f149a3485afc2b1b223746e075c"
server_key_iv: "c05f07089edc891fca8f94d6d87072f5"
server_key_version: "encryptionkey@0"
I20260620 06:15:40.685690 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "f13093324e444e9aa27a429e2ca99e4d"
format_stamp: "Formatted at 2026-06-20 06:15:40 on dist-test-slave-zt01"
server_key: "e0c39f149a3485afc2b1b223746e075c"
server_key_iv: "c05f07089edc891fca8f94d6d87072f5"
server_key_version: "encryptionkey@0"
I20260620 06:15:40.688983 3510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:15:40.691020 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:40.692139 3510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:15:40.692261 3510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "f13093324e444e9aa27a429e2ca99e4d"
format_stamp: "Formatted at 2026-06-20 06:15:40 on dist-test-slave-zt01"
server_key: "e0c39f149a3485afc2b1b223746e075c"
server_key_iv: "c05f07089edc891fca8f94d6d87072f5"
server_key_version: "encryptionkey@0"
I20260620 06:15:40.692355 3510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:40.696511 3510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:40.698985 3510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:40.699208 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:40.705621 3510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41791
I20260620 06:15:40.705634 3577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41791 every 8 connection(s)
I20260620 06:15:40.706463 3510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:40.708371 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3510
I20260620 06:15:40.708523 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:40.708802 15174 external_mini_cluster.cc:1428] Setting key cae9b53eb01eaf85e89b98095e442d76
I20260620 06:15:40.709115 3578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:40.713948 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:40.710039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44349 (local address 127.0.0.1:41791)
0620 06:15:40.710431 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:40.710450 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:40.710489 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:40.711276 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:40.711298 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:40.711635 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:40.711978 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:40.712670 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:40.712675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:40.712689 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:40.712701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:40.712759 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:40.712786 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:40.712934 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:40.713008 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 06:15:40.713138 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260620 06:15:40.715776 3578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d: Bootstrap starting.
I20260620 06:15:40.716543 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:40.717983 3578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:40.718796 3578 log.cc:824] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:40.720153 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:40.718425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38955 (local address 127.0.0.1:41791)
0620 06:15:40.718546 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:40.718550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:40.718766 (+ 216us) server_negotiation.cc:407] Connection header received
0620 06:15:40.718875 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:40.718877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:40.718933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:40.719015 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:40.719773 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:40.719776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:40.719779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:40.719789 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:40.719809 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:40.719818 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:40.719887 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:40.719988 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 06:15:40.720014 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:15:40.721029 3578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d: No bootstrap required, opened a new log
I20260620 06:15:40.724298 3578 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13093324e444e9aa27a429e2ca99e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41791 } }
I20260620 06:15:40.724512 3578 raft_consensus.cc:374] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:40.724568 3578 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13093324e444e9aa27a429e2ca99e4d, State: Initialized, Role: FOLLOWER
I20260620 06:15:40.724985 3578 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13093324e444e9aa27a429e2ca99e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41791 } }
I20260620 06:15:40.725135 3578 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:40.725199 3578 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:40.725291 3578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:40.726070 3578 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13093324e444e9aa27a429e2ca99e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41791 } }
I20260620 06:15:40.726384 3578 leader_election.cc:302] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13093324e444e9aa27a429e2ca99e4d; no voters:
I20260620 06:15:40.726734 3578 leader_election.cc:288] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:40.726783 3588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:40.726979 3588 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [term 1 LEADER]: Becoming Leader. State: Replica: f13093324e444e9aa27a429e2ca99e4d, State: Running, Role: LEADER
I20260620 06:15:40.727327 3588 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13093324e444e9aa27a429e2ca99e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41791 } }
I20260620 06:15:40.727918 3578 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:40.728871 3589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13093324e444e9aa27a429e2ca99e4d. Latest consensus state: current_term: 1 leader_uuid: "f13093324e444e9aa27a429e2ca99e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13093324e444e9aa27a429e2ca99e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41791 } } }
I20260620 06:15:40.729001 3589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:40.729316 3590 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f13093324e444e9aa27a429e2ca99e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13093324e444e9aa27a429e2ca99e4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41791 } } }
I20260620 06:15:40.729385 3597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:40.729462 3590 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:40.732519 3597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:40.737897 3597 catalog_manager.cc:1347] Generated new cluster ID: dc3c5fcb6ee44387b5548924a1b50aaa
I20260620 06:15:40.737958 3597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:40.761592 3597 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:40.761893 3597 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:40.777863 3597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f13093324e444e9aa27a429e2ca99e4d: Generated new TSK 0
I20260620 06:15:40.778530 3597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:40.782387 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3510
2026-06-20T06:15:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:40 dist-test-slave-zt01 krb5kdc[3609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:40 dist-test-slave-zt01 krb5kdc[3609](info): set up 2 sockets
Jun 20 06:15:40 dist-test-slave-zt01 krb5kdc[3609](info): commencing operation
krb5kdc: starting...
W20260620 06:15:41.819332 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:41 dist-test-slave-zt01 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:41.938265 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:42.042374 3622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:42.042588 3622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:42.042626 3622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:42.046443 3622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:42.046491 3622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:42.046511 3622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:42.046527 3622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:42.046541 3622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:42.051461 3622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:42.052263 3622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:42.053081 3622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:42.060971 3628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:42.061007 3627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:42.061098 3622 server_base.cc:1034] running on GCE node
W20260620 06:15:42.061251 3630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:42.061722 3622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:42.062717 3622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:42.063872 3622 hybrid_clock.cc:648] HybridClock initialized: now 1781936142063847 us; error 36 us; skew 500 ppm
Jun 20 06:15:42 dist-test-slave-zt01 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:42.066542 3622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:42.067461 3622 webserver.cc:460] Webserver started at http://127.0.0.1:34963/ using document root <none> and password file <none>
I20260620 06:15:42.067968 3622 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:42.068055 3622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:42.068279 3622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:42.070111 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "4fbf8236b21c4a15ab43648bc448c730"
format_stamp: "Formatted at 2026-06-20 06:15:42 on dist-test-slave-zt01"
server_key: "59fcab9b46556c06899b3fd595f5d1ee"
server_key_iv: "cda36a8fc84888824864360e18284b37"
server_key_version: "encryptionkey@0"
I20260620 06:15:42.070654 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "4fbf8236b21c4a15ab43648bc448c730"
format_stamp: "Formatted at 2026-06-20 06:15:42 on dist-test-slave-zt01"
server_key: "59fcab9b46556c06899b3fd595f5d1ee"
server_key_iv: "cda36a8fc84888824864360e18284b37"
server_key_version: "encryptionkey@0"
I20260620 06:15:42.074041 3622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:15:42.076362 3637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:42.077489 3622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:42.077628 3622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "4fbf8236b21c4a15ab43648bc448c730"
format_stamp: "Formatted at 2026-06-20 06:15:42 on dist-test-slave-zt01"
server_key: "59fcab9b46556c06899b3fd595f5d1ee"
server_key_iv: "cda36a8fc84888824864360e18284b37"
server_key_version: "encryptionkey@0"
I20260620 06:15:42.077734 3622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:42.101774 3622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:42.104905 3622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:42.105094 3622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:42.112654 3622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44383
I20260620 06:15:42.112680 3689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44383 every 8 connection(s)
I20260620 06:15:42.113600 3622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:42.116168 3690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:42.121232 3690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730: Bootstrap starting.
I20260620 06:15:42.123361 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3622
I20260620 06:15:42.123492 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:42.123572 3690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:42.123812 15174 external_mini_cluster.cc:1428] Setting key 73d681b16c7f462ca3b115ffbfdffbc4
I20260620 06:15:42.124289 3690 log.cc:824] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:42.126231 3690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730: No bootstrap required, opened a new log
I20260620 06:15:42.129467 3690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fbf8236b21c4a15ab43648bc448c730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260620 06:15:42.129698 3690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:42.129793 3690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbf8236b21c4a15ab43648bc448c730, State: Initialized, Role: FOLLOWER
I20260620 06:15:42.130271 3690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fbf8236b21c4a15ab43648bc448c730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260620 06:15:42.130422 3690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:42.130511 3690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:42.130616 3690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:42.131671 3690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fbf8236b21c4a15ab43648bc448c730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260620 06:15:42.132000 3690 leader_election.cc:302] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fbf8236b21c4a15ab43648bc448c730; no voters:
I20260620 06:15:42.132333 3690 leader_election.cc:288] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:42.132371 3695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:42.132552 3695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [term 1 LEADER]: Becoming Leader. State: Replica: 4fbf8236b21c4a15ab43648bc448c730, State: Running, Role: LEADER
I20260620 06:15:42.132886 3695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fbf8236b21c4a15ab43648bc448c730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
Jun 20 06:15:42 dist-test-slave-zt01 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:15:42.133494 3690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:42.136168 3696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fbf8236b21c4a15ab43648bc448c730. Latest consensus state: current_term: 1 leader_uuid: "4fbf8236b21c4a15ab43648bc448c730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fbf8236b21c4a15ab43648bc448c730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } } }
I20260620 06:15:42.136400 3696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:42.136289 3697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fbf8236b21c4a15ab43648bc448c730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fbf8236b21c4a15ab43648bc448c730" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } } }
I20260620 06:15:42.136662 3697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730 [sys.catalog]: This master's current role is: LEADER
W20260620 06:15:42.139005 3710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 06:15:42.139020 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:42.124845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:43599 (local address 127.0.0.1:44383)
0620 06:15:42.125202 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:42.125219 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:42.125261 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:15:42.125849 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:42.125864 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:42.126155 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:42.126516 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:42.127358 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:42.128305 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:42.129646 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:42.130621 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:42.130860 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:42.133926 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:42.133935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:42.133951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:42.133979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:42.136084 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:42.136639 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:42.136647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:42.136652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:42.136839 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:42.137177 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:42.137180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:42.137182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:42.137531 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:42.137765 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:42.138033 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 06:15:42.138257 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":118,"threads_started":1}
W20260620 06:15:42.139114 3710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:42.139400 3711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:42.140231 3711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:42.142221 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:42.146430 3711 catalog_manager.cc:1347] Generated new cluster ID: 423c52d0e57947ce8a1d36cf7a565d56
I20260620 06:15:42.146517 3711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:42.153146 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:42.144834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:40547 (local address 127.0.0.1:44383)
0620 06:15:42.144953 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:42.144957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:42.144972 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:15:42.145068 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:42.145070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:42.145102 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:42.145200 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:42.145876 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:42.146604 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:42.147730 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:42.148067 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:42.148143 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:42.149708 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:42.149711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:42.149714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:42.149735 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:42.151263 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:42.151757 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:42.151760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:42.151762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:42.151829 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:42.152126 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:42.152128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:42.152130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:42.152303 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:42.152423 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:42.152935 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 06:15:42.153011 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:15:42.174791 3711 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:42.176035 3711 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:42.188515 3711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fbf8236b21c4a15ab43648bc448c730: Generated new TSK 0
I20260620 06:15:42.189240 3711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:42.197100 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3622
W20260620 06:15:42.202355 3607 connection.cc:535] client connection to 127.0.0.1:44383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44383): Connection reset by peer (error 104)
2026-06-20T06:15:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:42 dist-test-slave-zt01 krb5kdc[3721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:42 dist-test-slave-zt01 krb5kdc[3721](info): set up 2 sockets
Jun 20 06:15:42 dist-test-slave-zt01 krb5kdc[3721](info): commencing operation
krb5kdc: starting...
W20260620 06:15:43.231201 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:43 dist-test-slave-zt01 krb5kdc[3721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:43.346158 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:43.449230 3734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:43.449512 3734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:43.449584 3734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:43.453552 3734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:43.453644 3734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:43.453681 3734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:43.453739 3734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:43.453773 3734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:43.458781 3734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:43.459756 3734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:43.460678 3734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:43.469164 3739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:43.469311 3740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:43.469342 3742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:43.469466 3734 server_base.cc:1034] running on GCE node
I20260620 06:15:43.470021 3734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:43.471035 3734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:43.472247 3734 hybrid_clock.cc:648] HybridClock initialized: now 1781936143472230 us; error 29 us; skew 500 ppm
Jun 20 06:15:43 dist-test-slave-zt01 krb5kdc[3721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:43.474792 3734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:43.475679 3734 webserver.cc:460] Webserver started at http://127.0.0.1:42091/ using document root <none> and password file <none>
I20260620 06:15:43.476159 3734 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:43.476215 3734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:43.476369 3734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:43.478178 3734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "327da4e01b49433f9843bae1b9f056c2"
format_stamp: "Formatted at 2026-06-20 06:15:43 on dist-test-slave-zt01"
server_key: "50daa999c4ddc3ac8e8c2cdc7a514d20"
server_key_iv: "3e17c802ee74df702537a1fad2051bd8"
server_key_version: "encryptionkey@0"
I20260620 06:15:43.478648 3734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "327da4e01b49433f9843bae1b9f056c2"
format_stamp: "Formatted at 2026-06-20 06:15:43 on dist-test-slave-zt01"
server_key: "50daa999c4ddc3ac8e8c2cdc7a514d20"
server_key_iv: "3e17c802ee74df702537a1fad2051bd8"
server_key_version: "encryptionkey@0"
I20260620 06:15:43.482033 3734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:43.484441 3749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:43.485605 3734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:43.485766 3734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "327da4e01b49433f9843bae1b9f056c2"
format_stamp: "Formatted at 2026-06-20 06:15:43 on dist-test-slave-zt01"
server_key: "50daa999c4ddc3ac8e8c2cdc7a514d20"
server_key_iv: "3e17c802ee74df702537a1fad2051bd8"
server_key_version: "encryptionkey@0"
I20260620 06:15:43.485893 3734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:43.500988 3734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:43.504104 3734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:43.504307 3734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:43.511047 3734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40515
I20260620 06:15:43.511101 3801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40515 every 8 connection(s)
I20260620 06:15:43.511924 3734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:43.514459 3802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:43.519784 3802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2: Bootstrap starting.
I20260620 06:15:43.521363 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3734
I20260620 06:15:43.521517 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:43.521811 15174 external_mini_cluster.cc:1428] Setting key 7af083b3eef7e986a4a606f6507b670a
I20260620 06:15:43.522353 3802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:43.523238 3802 log.cc:824] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:43.525161 3802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2: No bootstrap required, opened a new log
I20260620 06:15:43.529507 3802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327da4e01b49433f9843bae1b9f056c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260620 06:15:43.529763 3802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:43.529824 3802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327da4e01b49433f9843bae1b9f056c2, State: Initialized, Role: FOLLOWER
I20260620 06:15:43.530262 3802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327da4e01b49433f9843bae1b9f056c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260620 06:15:43.530396 3802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:43.530493 3802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:43.530624 3802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:43.531134 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:43.522888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:44019 (local address 127.0.0.1:40515)
0620 06:15:43.523335 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:43.523357 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:43.523396 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:43.524194 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:43.524224 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:43.524524 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:43.524805 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:43.525097 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:43.526092 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:43.527390 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:43.527855 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:43.527980 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:43.529108 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:43.529113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:43.529125 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:43.529490 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:43.529541 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:43.529563 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:43.529665 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:43.529981 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 06:15:43.530111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20260620 06:15:43.531630 3802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327da4e01b49433f9843bae1b9f056c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260620 06:15:43.531962 3802 leader_election.cc:302] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327da4e01b49433f9843bae1b9f056c2; no voters:
I20260620 06:15:43.532332 3802 leader_election.cc:288] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:43.532435 3807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:43.532639 3807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [term 1 LEADER]: Becoming Leader. State: Replica: 327da4e01b49433f9843bae1b9f056c2, State: Running, Role: LEADER
I20260620 06:15:43.533231 3807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327da4e01b49433f9843bae1b9f056c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } }
I20260620 06:15:43.533620 3802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:43.534103 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:43.534358 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327da4e01b49433f9843bae1b9f056c2. Latest consensus state: current_term: 1 leader_uuid: "327da4e01b49433f9843bae1b9f056c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327da4e01b49433f9843bae1b9f056c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } } }
I20260620 06:15:43.534458 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:43.535131 3808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "327da4e01b49433f9843bae1b9f056c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327da4e01b49433f9843bae1b9f056c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40515 } } }
I20260620 06:15:43.535214 3808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:43.536232 3815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:43.539479 3815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:43.541265 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:43.536577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:35113 (local address 127.0.0.1:40515)
0620 06:15:43.536714 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:43.536721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:43.536996 (+ 275us) server_negotiation.cc:407] Connection header received
0620 06:15:43.537101 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:43.537104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:43.537153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:43.537221 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:43.537533 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:43.538245 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:43.539399 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:43.539734 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:43.539851 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:43.540722 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:43.540725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:43.540728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:43.540741 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:43.540777 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:43.540791 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:43.540890 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:43.541083 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 06:15:43.541120 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:15:43.544807 3815 catalog_manager.cc:1347] Generated new cluster ID: 93cb0c6e2fb841c1b7f4e01da8b71c23
I20260620 06:15:43.544871 3815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:43.564249 3815 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:43.565300 3815 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:43.570698 3815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 327da4e01b49433f9843bae1b9f056c2: Generated new TSK 0
I20260620 06:15:43.571316 3815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:43.579381 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3734
W20260620 06:15:43.585932 3719 connection.cc:535] client connection to 127.0.0.1:40515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40515): Connection reset by peer (error 104)
2026-06-20T06:15:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:43 dist-test-slave-zt01 krb5kdc[3833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:43 dist-test-slave-zt01 krb5kdc[3833](info): set up 2 sockets
Jun 20 06:15:43 dist-test-slave-zt01 krb5kdc[3833](info): commencing operation
krb5kdc: starting...
W20260620 06:15:44.627193 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:44 dist-test-slave-zt01 krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:44.739374 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:44.842142 3846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:44.842398 3846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:44.842468 3846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:44.846431 3846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:44.846511 3846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:44.846553 3846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:44.846611 3846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:44.846648 3846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:44.851598 3846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:44.852485 3846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:44.853677 3846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:44.861785 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:44.861933 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:44.862008 3846 server_base.cc:1034] running on GCE node
W20260620 06:15:44.861873 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:44.862584 3846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:44.863628 3846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:44.864799 3846 hybrid_clock.cc:648] HybridClock initialized: now 1781936144864779 us; error 29 us; skew 500 ppm
Jun 20 06:15:44 dist-test-slave-zt01 krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:44.873835 3846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:44.874619 3846 webserver.cc:460] Webserver started at http://127.0.0.1:44251/ using document root <none> and password file <none>
I20260620 06:15:44.875197 3846 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:44.875262 3846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:44.875454 3846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:44.877310 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "039cdd17911d44ecb7a249e358d16df2"
format_stamp: "Formatted at 2026-06-20 06:15:44 on dist-test-slave-zt01"
server_key: "7e6f3f9a687d158e4c27639ed18d214e"
server_key_iv: "ca860273a4950f52c5af97c5dbda333c"
server_key_version: "encryptionkey@0"
I20260620 06:15:44.877830 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "039cdd17911d44ecb7a249e358d16df2"
format_stamp: "Formatted at 2026-06-20 06:15:44 on dist-test-slave-zt01"
server_key: "7e6f3f9a687d158e4c27639ed18d214e"
server_key_iv: "ca860273a4950f52c5af97c5dbda333c"
server_key_version: "encryptionkey@0"
I20260620 06:15:44.881220 3846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:15:44.883477 3861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:44.884637 3846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:44.884807 3846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "039cdd17911d44ecb7a249e358d16df2"
format_stamp: "Formatted at 2026-06-20 06:15:44 on dist-test-slave-zt01"
server_key: "7e6f3f9a687d158e4c27639ed18d214e"
server_key_iv: "ca860273a4950f52c5af97c5dbda333c"
server_key_version: "encryptionkey@0"
I20260620 06:15:44.884907 3846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:44.889151 3846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:44.892043 3846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:44.892225 3846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:44.898874 3846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36205
I20260620 06:15:44.898906 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36205 every 8 connection(s)
I20260620 06:15:44.899732 3846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:44.902124 3914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:44.904204 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3846
I20260620 06:15:44.904337 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:44.904595 15174 external_mini_cluster.cc:1428] Setting key 544515b042573fa4660d49b4fba70b64
I20260620 06:15:44.907867 3914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2: Bootstrap starting.
I20260620 06:15:44.909432 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:44.905891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:47091 (local address 127.0.0.1:36205)
0620 06:15:44.906282 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:44.906297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:44.906335 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:15:44.906950 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:44.906965 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:44.907281 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:44.907534 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:44.908279 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:44.908283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:44.908293 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:44.908302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:44.908353 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:44.908371 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:44.908473 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:44.908594 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 06:15:44.908819 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":130,"threads_started":1}
I20260620 06:15:44.910125 3914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:44.910820 3914 log.cc:824] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:44.911916 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:44.912741 3914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2: No bootstrap required, opened a new log
I20260620 06:15:44.915656 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:44.913850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:38605 (local address 127.0.0.1:36205)
0620 06:15:44.913972 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:44.913979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:44.914167 (+ 188us) server_negotiation.cc:407] Connection header received
0620 06:15:44.914321 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:44.914329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:44.914355 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:44.914435 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:44.915263 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:44.915267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:44.915270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:44.915281 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:44.915313 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:44.915324 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:44.915417 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:44.915493 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 06:15:44.915530 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:15:44.916675 3914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039cdd17911d44ecb7a249e358d16df2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260620 06:15:44.916865 3914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:44.916898 3914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039cdd17911d44ecb7a249e358d16df2, State: Initialized, Role: FOLLOWER
I20260620 06:15:44.917268 3914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039cdd17911d44ecb7a249e358d16df2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260620 06:15:44.917379 3914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:44.917418 3914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:44.917484 3914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:44.918236 3914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039cdd17911d44ecb7a249e358d16df2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260620 06:15:44.918500 3914 leader_election.cc:302] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 039cdd17911d44ecb7a249e358d16df2; no voters:
I20260620 06:15:44.918769 3914 leader_election.cc:288] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:44.918833 3924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:44.919015 3924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [term 1 LEADER]: Becoming Leader. State: Replica: 039cdd17911d44ecb7a249e358d16df2, State: Running, Role: LEADER
I20260620 06:15:44.919380 3924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039cdd17911d44ecb7a249e358d16df2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } }
I20260620 06:15:44.919957 3914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:44.921056 3926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 039cdd17911d44ecb7a249e358d16df2. Latest consensus state: current_term: 1 leader_uuid: "039cdd17911d44ecb7a249e358d16df2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039cdd17911d44ecb7a249e358d16df2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } } }
I20260620 06:15:44.921070 3925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "039cdd17911d44ecb7a249e358d16df2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039cdd17911d44ecb7a249e358d16df2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36205 } } }
I20260620 06:15:44.921232 3925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:44.921231 3926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:44.921734 3933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:44.924427 3933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:44.929448 3933 catalog_manager.cc:1347] Generated new cluster ID: 258fc6f6c1964f97a736c003e28aa191
I20260620 06:15:44.929535 3933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:44.939636 3933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:44.939919 3933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:44.950081 3933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 039cdd17911d44ecb7a249e358d16df2: Generated new TSK 0
I20260620 06:15:44.950796 3933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:44.958551 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3846
2026-06-20T06:15:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:45 dist-test-slave-zt01 krb5kdc[3945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:45 dist-test-slave-zt01 krb5kdc[3945](info): set up 2 sockets
Jun 20 06:15:45 dist-test-slave-zt01 krb5kdc[3945](info): commencing operation
krb5kdc: starting...
W20260620 06:15:45.999188 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:46 dist-test-slave-zt01 krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:46.113857 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:46.219177 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:46.219431 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:46.219502 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:46.223399 3958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:46.223483 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:46.223520 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:46.223577 3958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:46.223615 3958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:46.228616 3958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:46.229539 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:46.230466 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:46.238346 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:46.238366 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:46.238507 3958 server_base.cc:1034] running on GCE node
W20260620 06:15:46.238356 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:46.239228 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:46.240237 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:46.241443 3958 hybrid_clock.cc:648] HybridClock initialized: now 1781936146241414 us; error 42 us; skew 500 ppm
Jun 20 06:15:46 dist-test-slave-zt01 krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:46.244252 3958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:46.245164 3958 webserver.cc:460] Webserver started at http://127.0.0.1:35341/ using document root <none> and password file <none>
I20260620 06:15:46.245683 3958 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:46.245779 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:46.245991 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:46.247957 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "8808c06d4ec146b88ac6652243a52e1c"
format_stamp: "Formatted at 2026-06-20 06:15:46 on dist-test-slave-zt01"
server_key: "616716999167288acda20f7f10f6f94a"
server_key_iv: "066184c47b518e14dc38ef92736e7e43"
server_key_version: "encryptionkey@0"
I20260620 06:15:46.248489 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "8808c06d4ec146b88ac6652243a52e1c"
format_stamp: "Formatted at 2026-06-20 06:15:46 on dist-test-slave-zt01"
server_key: "616716999167288acda20f7f10f6f94a"
server_key_iv: "066184c47b518e14dc38ef92736e7e43"
server_key_version: "encryptionkey@0"
I20260620 06:15:46.251940 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:15:46.254165 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:46.255306 3958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:46.255441 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "8808c06d4ec146b88ac6652243a52e1c"
format_stamp: "Formatted at 2026-06-20 06:15:46 on dist-test-slave-zt01"
server_key: "616716999167288acda20f7f10f6f94a"
server_key_iv: "066184c47b518e14dc38ef92736e7e43"
server_key_version: "encryptionkey@0"
I20260620 06:15:46.255546 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:46.275871 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:46.278364 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:46.278548 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:46.285575 3958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40849
I20260620 06:15:46.285632 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40849 every 8 connection(s)
I20260620 06:15:46.286466 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:46.289072 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:46.289259 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 3958
I20260620 06:15:46.289412 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:46.289685 15174 external_mini_cluster.cc:1428] Setting key 4b4d3cb3bb4d02a0e78825553adcd360
I20260620 06:15:46.295749 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c: Bootstrap starting.
I20260620 06:15:46.297992 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c: Neither blocks nor log segments found. Creating new log.
Jun 20 06:15:46 dist-test-slave-zt01 krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:15:46.298681 4026 log.cc:824] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:46.300577 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c: No bootstrap required, opened a new log
I20260620 06:15:46.303516 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:46.290960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:54597 (local address 127.0.0.1:40849)
0620 06:15:46.291422 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:46.291439 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:46.291478 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:46.292232 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:46.292261 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:46.292655 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:46.292998 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:46.293612 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:46.295112 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:46.296007 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:46.296649 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:46.296821 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:46.299217 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:46.299221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:46.299232 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:46.299256 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:46.300871 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:46.301358 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:46.301363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:46.301367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:46.301464 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:46.301790 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:46.301793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:46.301795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:46.302128 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:46.302342 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:46.302643 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 06:15:46.302836 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":213,"threads_started":1}
I20260620 06:15:46.303994 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808c06d4ec146b88ac6652243a52e1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260620 06:15:46.304207 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:46.304896 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8808c06d4ec146b88ac6652243a52e1c, State: Initialized, Role: FOLLOWER
I20260620 06:15:46.305387 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808c06d4ec146b88ac6652243a52e1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260620 06:15:46.305536 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:46.305617 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:46.305737 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:46.306066 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:46.306730 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808c06d4ec146b88ac6652243a52e1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260620 06:15:46.307101 4026 leader_election.cc:302] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8808c06d4ec146b88ac6652243a52e1c; no voters:
I20260620 06:15:46.307444 4026 leader_election.cc:288] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:46.307546 4032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:46.307753 4032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [term 1 LEADER]: Becoming Leader. State: Replica: 8808c06d4ec146b88ac6652243a52e1c, State: Running, Role: LEADER
I20260620 06:15:46.308108 4032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808c06d4ec146b88ac6652243a52e1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260620 06:15:46.308676 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:46.309753 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8808c06d4ec146b88ac6652243a52e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808c06d4ec146b88ac6652243a52e1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260620 06:15:46.309878 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:46.310199 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8808c06d4ec146b88ac6652243a52e1c. Latest consensus state: current_term: 1 leader_uuid: "8808c06d4ec146b88ac6652243a52e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808c06d4ec146b88ac6652243a52e1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260620 06:15:46.310261 4045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:46.310328 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:46.313241 4045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:46.318686 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:46.308271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:50905 (local address 127.0.0.1:40849)
0620 06:15:46.308512 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:46.308516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:46.311011 (+ 2495us) server_negotiation.cc:407] Connection header received
0620 06:15:46.311158 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:46.311161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:46.311203 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:46.311298 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:46.311963 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:46.312458 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:46.313243 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:46.313539 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:46.313646 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:46.314955 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:46.314958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:46.314961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:46.314984 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:46.316591 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:46.317180 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:46.317184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:46.317186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:46.317271 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:46.317686 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:46.317690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:46.317692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:46.317903 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:46.318017 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:46.318443 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 06:15:46.318539 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260620 06:15:46.318691 4045 catalog_manager.cc:1347] Generated new cluster ID: 635a372d1a5c414885e513cd9d61d11a
I20260620 06:15:46.319056 4045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:46.331694 4045 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:46.332835 4045 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:46.342116 4045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8808c06d4ec146b88ac6652243a52e1c: Generated new TSK 0
I20260620 06:15:46.342765 4045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:46.346405 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 3958
W20260620 06:15:46.352216 3943 connection.cc:535] client connection to 127.0.0.1:40849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40849): Connection reset by peer (error 104)
2026-06-20T06:15:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:46 dist-test-slave-zt01 krb5kdc[4057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:46 dist-test-slave-zt01 krb5kdc[4057](info): set up 2 sockets
Jun 20 06:15:46 dist-test-slave-zt01 krb5kdc[4057](info): commencing operation
krb5kdc: starting...
W20260620 06:15:47.455282 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:47 dist-test-slave-zt01 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:47.571398 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:47.676420 4070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:47.676635 4070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:47.676674 4070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:47.680536 4070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:47.680583 4070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:47.680603 4070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:47.680619 4070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:47.680677 4070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:47.685695 4070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40003
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:47.686548 4070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:47.687466 4070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:47.695363 4078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:47.695349 4076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:47.695423 4070 server_base.cc:1034] running on GCE node
W20260620 06:15:47.695343 4075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:47.696136 4070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:47.697070 4070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:47.698257 4070 hybrid_clock.cc:648] HybridClock initialized: now 1781936147698242 us; error 40 us; skew 500 ppm
Jun 20 06:15:47 dist-test-slave-zt01 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:47.700886 4070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:47.701697 4070 webserver.cc:460] Webserver started at http://127.0.0.1:34259/ using document root <none> and password file <none>
I20260620 06:15:47.702234 4070 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:47.702325 4070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:47.702539 4070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:47.704427 4070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "f0e976d7f7e9405485dba0cd3611c303"
format_stamp: "Formatted at 2026-06-20 06:15:47 on dist-test-slave-zt01"
server_key: "048c6696d15d16242ecaac57ace9d2f0"
server_key_iv: "b9e169360b985d6c4e7c8a022e7b6ea9"
server_key_version: "encryptionkey@0"
I20260620 06:15:47.704944 4070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "f0e976d7f7e9405485dba0cd3611c303"
format_stamp: "Formatted at 2026-06-20 06:15:47 on dist-test-slave-zt01"
server_key: "048c6696d15d16242ecaac57ace9d2f0"
server_key_iv: "b9e169360b985d6c4e7c8a022e7b6ea9"
server_key_version: "encryptionkey@0"
I20260620 06:15:47.708339 4070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:15:47.710440 4085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:47.711555 4070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:15:47.711687 4070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "f0e976d7f7e9405485dba0cd3611c303"
format_stamp: "Formatted at 2026-06-20 06:15:47 on dist-test-slave-zt01"
server_key: "048c6696d15d16242ecaac57ace9d2f0"
server_key_iv: "b9e169360b985d6c4e7c8a022e7b6ea9"
server_key_version: "encryptionkey@0"
I20260620 06:15:47.711817 4070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:47.751362 4070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:47.753962 4070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:47.754151 4070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:47.761200 4070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40003
I20260620 06:15:47.761217 4137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40003 every 8 connection(s)
I20260620 06:15:47.762048 4070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:47.764619 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:47.766700 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4070
I20260620 06:15:47.766841 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:47.767153 15174 external_mini_cluster.cc:1428] Setting key 2ea64cbcfb773c0e04e0867d86c3f8da
I20260620 06:15:47.770639 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303: Bootstrap starting.
I20260620 06:15:47.772958 4138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:47.773795 4138 log.cc:824] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:47.775921 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303: No bootstrap required, opened a new log
I20260620 06:15:47.776530 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:47.768440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:59563 (local address 127.0.0.1:40003)
0620 06:15:47.768820 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:47.768840 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:47.768879 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:15:47.769644 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:47.769664 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:47.770053 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:47.770389 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:47.770758 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:47.772170 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:47.773058 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:47.773721 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:47.773881 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:47.774674 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:47.774680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:47.774694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:47.775157 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:47.775221 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:47.775262 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:47.775386 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:47.775528 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 06:15:47.775640 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260620 06:15:47.779220 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:47.779886 4138 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e976d7f7e9405485dba0cd3611c303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40003 } }
I20260620 06:15:47.780083 4138 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:47.780148 4138 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e976d7f7e9405485dba0cd3611c303, State: Initialized, Role: FOLLOWER
I20260620 06:15:47.780627 4138 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e976d7f7e9405485dba0cd3611c303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40003 } }
I20260620 06:15:47.780745 4138 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:47.780831 4138 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:47.780944 4138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:47.782023 4138 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e976d7f7e9405485dba0cd3611c303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40003 } }
I20260620 06:15:47.782352 4138 leader_election.cc:302] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e976d7f7e9405485dba0cd3611c303; no voters:
I20260620 06:15:47.782658 4138 leader_election.cc:288] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:47.782763 4148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:47.783000 4148 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [term 1 LEADER]: Becoming Leader. State: Replica: f0e976d7f7e9405485dba0cd3611c303, State: Running, Role: LEADER
I20260620 06:15:47.783360 4148 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e976d7f7e9405485dba0cd3611c303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40003 } }
I20260620 06:15:47.783833 4138 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:47.784519 4150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e976d7f7e9405485dba0cd3611c303. Latest consensus state: current_term: 1 leader_uuid: "f0e976d7f7e9405485dba0cd3611c303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e976d7f7e9405485dba0cd3611c303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40003 } } }
I20260620 06:15:47.784695 4150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:47.785022 4149 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e976d7f7e9405485dba0cd3611c303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e976d7f7e9405485dba0cd3611c303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40003 } } }
I20260620 06:15:47.785135 4149 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:47.786280 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:47.781176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:37317 (local address 127.0.0.1:40003)
0620 06:15:47.781300 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:47.781304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:47.781499 (+ 195us) server_negotiation.cc:407] Connection header received
0620 06:15:47.781600 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:47.781602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:47.781635 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:47.781697 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:47.781976 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:47.782550 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:47.783635 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:47.783998 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:47.784096 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:47.785587 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:47.785591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:47.785594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:47.785604 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:47.785631 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:47.785646 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:47.785736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:47.786091 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 06:15:47.786120 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 06:15:47.788930 4164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:47.789006 4164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:47.789108 4153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:47.789786 4153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:47.794684 4153 catalog_manager.cc:1347] Generated new cluster ID: 814fdbae03584820852d187c142867dc
I20260620 06:15:47.794742 4153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:47.809919 4153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:47.811041 4153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:47.822769 4153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e976d7f7e9405485dba0cd3611c303: Generated new TSK 0
I20260620 06:15:47.823479 4153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:47.834281 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4070
W20260620 06:15:47.839658 4055 connection.cc:535] client connection to 127.0.0.1:40003 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40003): Connection reset by peer (error 104)
2026-06-20T06:15:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:47 dist-test-slave-zt01 krb5kdc[4169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:47 dist-test-slave-zt01 krb5kdc[4169](info): set up 2 sockets
Jun 20 06:15:47 dist-test-slave-zt01 krb5kdc[4169](info): commencing operation
krb5kdc: starting...
W20260620 06:15:48.863621 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:48 dist-test-slave-zt01 krb5kdc[4169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:15:48.981653 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:49.090673 4182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:49.090898 4182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:49.090936 4182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:49.094798 4182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:49.094851 4182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:49.094869 4182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:49.094885 4182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:49.094931 4182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:49.100128 4182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:49.101001 4182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:49.101914 4182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:49.109761 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:49.109828 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:49.109787 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:49.109963 4182 server_base.cc:1034] running on GCE node
I20260620 06:15:49.110441 4182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:49.111567 4182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:49.112752 4182 hybrid_clock.cc:648] HybridClock initialized: now 1781936149112713 us; error 35 us; skew 500 ppm
Jun 20 06:15:49 dist-test-slave-zt01 krb5kdc[4169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:49.115418 4182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:15:49.116209 4182 webserver.cc:460] Webserver started at http://127.0.0.1:33003/ using document root <none> and password file <none>
I20260620 06:15:49.116712 4182 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:49.116798 4182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:49.117012 4182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:49.118882 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "b1be692f657f431fb430659fbaddb295"
format_stamp: "Formatted at 2026-06-20 06:15:49 on dist-test-slave-zt01"
server_key: "bfdfd101cd59b22dd9442ffce68a2eed"
server_key_iv: "651a9a4029e952d110fd33ba8b26cf46"
server_key_version: "encryptionkey@0"
I20260620 06:15:49.119513 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "b1be692f657f431fb430659fbaddb295"
format_stamp: "Formatted at 2026-06-20 06:15:49 on dist-test-slave-zt01"
server_key: "bfdfd101cd59b22dd9442ffce68a2eed"
server_key_iv: "651a9a4029e952d110fd33ba8b26cf46"
server_key_version: "encryptionkey@0"
I20260620 06:15:49.124327 4182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:15:49.127357 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:49.128809 4182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:15:49.128928 4182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "b1be692f657f431fb430659fbaddb295"
format_stamp: "Formatted at 2026-06-20 06:15:49 on dist-test-slave-zt01"
server_key: "bfdfd101cd59b22dd9442ffce68a2eed"
server_key_iv: "651a9a4029e952d110fd33ba8b26cf46"
server_key_version: "encryptionkey@0"
I20260620 06:15:49.129030 4182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:49.133230 4182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:49.136062 4182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:49.136240 4182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:49.142700 4182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36083
I20260620 06:15:49.142768 4249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36083 every 8 connection(s)
I20260620 06:15:49.143594 4182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:49.146108 4250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:49.147198 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4182
I20260620 06:15:49.147331 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:49.147590 15174 external_mini_cluster.cc:1428] Setting key 95f5fb2be7739807f36e05d6cca004c7
I20260620 06:15:49.152405 4250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295: Bootstrap starting.
I20260620 06:15:49.152652 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:49.148829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:45121 (local address 127.0.0.1:36083)
0620 06:15:49.149242 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:49.149260 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:49.149298 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:15:49.150090 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:49.150110 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:49.150435 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:49.150779 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:49.151400 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:49.151406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:49.151419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:49.151429 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:49.151487 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:49.151516 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:49.151649 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:49.151713 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:15:49.151859 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260620 06:15:49.154747 4250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:49.155210 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:49.155555 4250 log.cc:824] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:49.157622 4250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295: No bootstrap required, opened a new log
I20260620 06:15:49.158790 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:49.157260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:49153 (local address 127.0.0.1:36083)
0620 06:15:49.157367 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:49.157370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:49.157600 (+ 230us) server_negotiation.cc:407] Connection header received
0620 06:15:49.157723 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:49.157726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:49.157763 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:49.157825 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:49.158433 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:49.158436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:49.158437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:49.158447 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:49.158466 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:49.158476 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:49.158540 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:49.158643 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 06:15:49.158672 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:15:49.161473 4250 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1be692f657f431fb430659fbaddb295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260620 06:15:49.161688 4250 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:49.161746 4250 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1be692f657f431fb430659fbaddb295, State: Initialized, Role: FOLLOWER
I20260620 06:15:49.162189 4250 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1be692f657f431fb430659fbaddb295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260620 06:15:49.162335 4250 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:49.162400 4250 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:49.162511 4250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:49.163388 4250 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1be692f657f431fb430659fbaddb295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260620 06:15:49.163693 4250 leader_election.cc:302] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1be692f657f431fb430659fbaddb295; no voters:
I20260620 06:15:49.163960 4250 leader_election.cc:288] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:49.164070 4260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:49.164302 4260 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [term 1 LEADER]: Becoming Leader. State: Replica: b1be692f657f431fb430659fbaddb295, State: Running, Role: LEADER
I20260620 06:15:49.164620 4260 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1be692f657f431fb430659fbaddb295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260620 06:15:49.165076 4250 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:49.166154 4261 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1be692f657f431fb430659fbaddb295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1be692f657f431fb430659fbaddb295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } } }
I20260620 06:15:49.166276 4261 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:49.166602 4262 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1be692f657f431fb430659fbaddb295. Latest consensus state: current_term: 1 leader_uuid: "b1be692f657f431fb430659fbaddb295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1be692f657f431fb430659fbaddb295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } } }
I20260620 06:15:49.166743 4262 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:49.166690 4269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:49.169788 4269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:49.174999 4269 catalog_manager.cc:1347] Generated new cluster ID: 3028c1e213fc4d389bd96fcdf907f56f
I20260620 06:15:49.175100 4269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:49.195243 4269 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:49.195508 4269 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:49.205761 4269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1be692f657f431fb430659fbaddb295: Generated new TSK 0
I20260620 06:15:49.206363 4269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:49.216480 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4182
2026-06-20T06:15:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:49 dist-test-slave-zt01 krb5kdc[4281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:49 dist-test-slave-zt01 krb5kdc[4281](info): set up 2 sockets
Jun 20 06:15:49 dist-test-slave-zt01 krb5kdc[4281](info): commencing operation
krb5kdc: starting...
W20260620 06:15:50.255270 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:50 dist-test-slave-zt01 krb5kdc[4281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:50.370451 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34687
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:42915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:50.473735 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:50.473954 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:50.473994 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:50.477955 4294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:50.478012 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:50.478031 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:50.478047 4294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:50.478093 4294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:50.482986 4294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:42915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:34687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:50.483886 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:50.484755 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:50.492650 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:50.492761 4294 server_base.cc:1034] running on GCE node
W20260620 06:15:50.492848 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:50.492861 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:50.493433 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:50.494364 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:50.495548 4294 hybrid_clock.cc:648] HybridClock initialized: now 1781936150495520 us; error 41 us; skew 500 ppm
Jun 20 06:15:50 dist-test-slave-zt01 krb5kdc[4281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:50.502573 4294 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:50.503494 4294 webserver.cc:460] Webserver started at http://127.14.209.190:38833/ using document root <none> and password file <none>
I20260620 06:15:50.504222 4294 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:50.504341 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:50.504578 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:50.506534 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "c817f11a33e74f9a8a1b5114d1925057"
format_stamp: "Formatted at 2026-06-20 06:15:50 on dist-test-slave-zt01"
server_key: "7cb85568611e4232b3e2a57c95157e3f"
server_key_iv: "1fd2f16d3a8cc5a58036bf63f856f183"
server_key_version: "encryptionkey@0"
I20260620 06:15:50.507122 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "c817f11a33e74f9a8a1b5114d1925057"
format_stamp: "Formatted at 2026-06-20 06:15:50 on dist-test-slave-zt01"
server_key: "7cb85568611e4232b3e2a57c95157e3f"
server_key_iv: "1fd2f16d3a8cc5a58036bf63f856f183"
server_key_version: "encryptionkey@0"
I20260620 06:15:50.510504 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:15:50.512804 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:50.513912 4294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:50.514052 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "c817f11a33e74f9a8a1b5114d1925057"
format_stamp: "Formatted at 2026-06-20 06:15:50 on dist-test-slave-zt01"
server_key: "7cb85568611e4232b3e2a57c95157e3f"
server_key_iv: "1fd2f16d3a8cc5a58036bf63f856f183"
server_key_version: "encryptionkey@0"
I20260620 06:15:50.514156 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:50.533056 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:50.535634 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:50.535840 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:50.542567 4294 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:34687
I20260620 06:15:50.542584 4361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:34687 every 8 connection(s)
I20260620 06:15:50.543424 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:50.545651 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4294
I20260620 06:15:50.545799 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:50.545919 4362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:50.546113 15174 external_mini_cluster.cc:1428] Setting key 56927f424b34681899c88f56bf3f5415
I20260620 06:15:50.551676 4362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057: Bootstrap starting.
I20260620 06:15:50.553800 4362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057: Neither blocks nor log segments found. Creating new log.
Jun 20 06:15:50 dist-test-slave-zt01 krb5kdc[4281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:50.554649 4362 log.cc:824] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:50.556537 4362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057: No bootstrap required, opened a new log
I20260620 06:15:50.559387 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:50.547304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39741 (local address 127.14.209.190:34687)
0620 06:15:50.547714 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:50.547727 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:50.547758 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:15:50.548496 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:50.548511 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:50.548796 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:50.549117 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:50.549870 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:50.550790 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:50.551723 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:50.552227 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:50.552390 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:50.555021 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:50.555028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:50.555043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:50.555096 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:50.556718 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:50.557235 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:50.557239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:50.557243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:50.557340 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:50.557661 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:50.557663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:50.557664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:50.558036 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:50.558183 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:50.558488 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 06:15:50.558661 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260620 06:15:50.560426 4362 raft_consensus.cc:348] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c817f11a33e74f9a8a1b5114d1925057" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34687 } }
I20260620 06:15:50.560659 4362 raft_consensus.cc:374] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:50.560711 4362 raft_consensus.cc:729] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c817f11a33e74f9a8a1b5114d1925057, State: Initialized, Role: FOLLOWER
I20260620 06:15:50.561100 4362 consensus_queue.cc:260] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c817f11a33e74f9a8a1b5114d1925057" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34687 } }
I20260620 06:15:50.561269 4362 raft_consensus.cc:388] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:50.561350 4362 raft_consensus.cc:482] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:50.561441 4362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:50.562099 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:50.562451 4362 raft_consensus.cc:504] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c817f11a33e74f9a8a1b5114d1925057" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34687 } }
I20260620 06:15:50.562788 4362 leader_election.cc:302] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c817f11a33e74f9a8a1b5114d1925057; no voters:
I20260620 06:15:50.563125 4362 leader_election.cc:288] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:50.563249 4367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:50.563550 4367 raft_consensus.cc:686] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [term 1 LEADER]: Becoming Leader. State: Replica: c817f11a33e74f9a8a1b5114d1925057, State: Running, Role: LEADER
I20260620 06:15:50.563845 4367 consensus_queue.cc:237] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c817f11a33e74f9a8a1b5114d1925057" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34687 } }
I20260620 06:15:50.564294 4362 sys_catalog.cc:564] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:50.565160 4367 sys_catalog.cc:455] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c817f11a33e74f9a8a1b5114d1925057. Latest consensus state: current_term: 1 leader_uuid: "c817f11a33e74f9a8a1b5114d1925057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c817f11a33e74f9a8a1b5114d1925057" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34687 } } }
I20260620 06:15:50.565299 4367 sys_catalog.cc:458] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:50.565554 4369 sys_catalog.cc:455] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c817f11a33e74f9a8a1b5114d1925057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c817f11a33e74f9a8a1b5114d1925057" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 34687 } } }
I20260620 06:15:50.565657 4369 sys_catalog.cc:458] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:50.565689 4381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:50.568572 4381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:50.573961 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:50.564191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37301 (local address 127.14.209.190:34687)
0620 06:15:50.564288 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:50.564291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:50.564768 (+ 477us) server_negotiation.cc:407] Connection header received
0620 06:15:50.564874 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:50.564877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:50.564922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:50.565013 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:50.566526 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:50.567269 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:50.568286 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:50.568598 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:50.568705 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:50.570436 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:50.570440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:50.570442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:50.570464 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:50.572080 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:50.572532 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:50.572536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:50.572538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:50.572623 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:50.573041 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:50.573045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:50.573046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:50.573294 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:50.573442 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:50.573744 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 06:15:50.573832 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:15:50.574440 4381 catalog_manager.cc:1347] Generated new cluster ID: 622249453ba1449b8186f69c5433a82d
I20260620 06:15:50.574489 4381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:50.582096 4381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:50.583164 4381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:50.589672 4381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c817f11a33e74f9a8a1b5114d1925057: Generated new TSK 0
I20260620 06:15:50.590226 4381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:50.600337 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4294
W20260620 06:15:50.605522 4279 connection.cc:535] client connection to 127.14.209.190:34687 recv error: Network error: failed to read from TLS socket (remote: 127.14.209.190:34687): Connection reset by peer (error 104)
2026-06-20T06:15:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:50 dist-test-slave-zt01 krb5kdc[4393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:50 dist-test-slave-zt01 krb5kdc[4393](info): set up 2 sockets
Jun 20 06:15:50 dist-test-slave-zt01 krb5kdc[4393](info): commencing operation
krb5kdc: starting...
W20260620 06:15:51.639216 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:51 dist-test-slave-zt01 krb5kdc[4393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:51.752123 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39145
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:51.854945 4406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:51.855262 4406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:51.855348 4406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:51.859236 4406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:51.859315 4406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:51.859352 4406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:51.859411 4406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:51.859447 4406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:51.864480 4406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:39145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:51.865381 4406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:51.866321 4406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:51.874193 4414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:51.874194 4411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:51.874217 4412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:51.874264 4406 server_base.cc:1034] running on GCE node
I20260620 06:15:51.874964 4406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:51.875963 4406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:51.877123 4406 hybrid_clock.cc:648] HybridClock initialized: now 1781936151877096 us; error 41 us; skew 500 ppm
Jun 20 06:15:51 dist-test-slave-zt01 krb5kdc[4393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:51.879885 4406 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:51.880712 4406 webserver.cc:460] Webserver started at http://127.14.209.190:41535/ using document root <none> and password file <none>
I20260620 06:15:51.881206 4406 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:51.881263 4406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:51.881464 4406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:51.883412 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "c48451a60f8e4b7aa2048d20e2d00e19"
format_stamp: "Formatted at 2026-06-20 06:15:51 on dist-test-slave-zt01"
server_key: "2690725f7086b3a8218759a200ec89e2"
server_key_iv: "40d2a379115f430a64a4cc5908ca8680"
server_key_version: "encryptionkey@0"
I20260620 06:15:51.883941 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "c48451a60f8e4b7aa2048d20e2d00e19"
format_stamp: "Formatted at 2026-06-20 06:15:51 on dist-test-slave-zt01"
server_key: "2690725f7086b3a8218759a200ec89e2"
server_key_iv: "40d2a379115f430a64a4cc5908ca8680"
server_key_version: "encryptionkey@0"
I20260620 06:15:51.887389 4406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:51.889609 4421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:51.890736 4406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:51.890875 4406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "c48451a60f8e4b7aa2048d20e2d00e19"
format_stamp: "Formatted at 2026-06-20 06:15:51 on dist-test-slave-zt01"
server_key: "2690725f7086b3a8218759a200ec89e2"
server_key_iv: "40d2a379115f430a64a4cc5908ca8680"
server_key_version: "encryptionkey@0"
I20260620 06:15:51.890980 4406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:51.904706 4406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:51.907706 4406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:51.907899 4406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:51.914630 4406 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:39145
I20260620 06:15:51.914649 4473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:39145 every 8 connection(s)
I20260620 06:15:51.915542 4406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:51.916976 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4406
I20260620 06:15:51.917135 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:51.917410 15174 external_mini_cluster.cc:1428] Setting key 0cba58755aac99820bad73882ac6a3c8
I20260620 06:15:51.918391 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:51.924342 4474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19: Bootstrap starting.
I20260620 06:15:51.925917 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:51.918665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60253 (local address 127.14.209.190:39145)
0620 06:15:51.919095 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:51.919118 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:51.919150 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:15:51.919740 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:51.919753 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:51.920050 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:51.920318 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:51.920639 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:51.921564 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:51.922774 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:51.923320 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:51.923437 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:51.924269 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:51.924286 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:51.924296 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:51.924721 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:51.924774 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:51.924804 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:51.924910 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:51.925133 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 06:15:51.925265 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":141,"threads_started":1}
I20260620 06:15:51.926483 4474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:51.927333 4474 log.cc:824] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:51.928718 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:51.929244 4474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19: No bootstrap required, opened a new log
I20260620 06:15:51.933362 4474 raft_consensus.cc:348] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39145 } }
I20260620 06:15:51.933627 4474 raft_consensus.cc:374] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:51.933708 4474 raft_consensus.cc:729] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48451a60f8e4b7aa2048d20e2d00e19, State: Initialized, Role: FOLLOWER
I20260620 06:15:51.934193 4474 consensus_queue.cc:260] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39145 } }
I20260620 06:15:51.934389 4474 raft_consensus.cc:388] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:51.934470 4474 raft_consensus.cc:482] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:51.934595 4474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:51.934635 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:51.930614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47499 (local address 127.14.209.190:39145)
0620 06:15:51.930722 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:51.930726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:51.930867 (+ 141us) server_negotiation.cc:407] Connection header received
0620 06:15:51.930982 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:51.930984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:51.931016 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:51.931143 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:51.931436 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:51.931885 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:51.932985 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:51.933185 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:51.933251 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:51.934116 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:51.934118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:51.934120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:51.934130 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:51.934151 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:51.934160 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:51.934225 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:51.934423 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 06:15:51.934462 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:15:51.935583 4474 raft_consensus.cc:504] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39145 } }
I20260620 06:15:51.935915 4474 leader_election.cc:302] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c48451a60f8e4b7aa2048d20e2d00e19; no voters:
I20260620 06:15:51.936214 4474 leader_election.cc:288] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:51.936357 4484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:51.936583 4484 raft_consensus.cc:686] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [term 1 LEADER]: Becoming Leader. State: Replica: c48451a60f8e4b7aa2048d20e2d00e19, State: Running, Role: LEADER
I20260620 06:15:51.936942 4484 consensus_queue.cc:237] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39145 } }
I20260620 06:15:51.937386 4474 sys_catalog.cc:564] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:51.939167 4485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39145 } } }
I20260620 06:15:51.939286 4485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:51.939627 4486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c48451a60f8e4b7aa2048d20e2d00e19. Latest consensus state: current_term: 1 leader_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48451a60f8e4b7aa2048d20e2d00e19" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39145 } } }
I20260620 06:15:51.939621 4499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:51.940022 4486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:51.942245 4499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:51.946946 4499 catalog_manager.cc:1347] Generated new cluster ID: 87843d735ae24895b31e09988ff714df
I20260620 06:15:51.947005 4499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:51.953524 4499 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:51.954566 4499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:51.962009 4499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c48451a60f8e4b7aa2048d20e2d00e19: Generated new TSK 0
I20260620 06:15:51.962620 4499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:51.966517 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4406
W20260620 06:15:51.970832 4391 connection.cc:535] client connection to 127.14.209.190:39145 recv error: Network error: failed to read from TLS socket (remote: 127.14.209.190:39145): Connection reset by peer (error 104)
2026-06-20T06:15:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:52 dist-test-slave-zt01 krb5kdc[4505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:52 dist-test-slave-zt01 krb5kdc[4505](info): set up 2 sockets
Jun 20 06:15:52 dist-test-slave-zt01 krb5kdc[4505](info): commencing operation
krb5kdc: starting...
W20260620 06:15:52.995220 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:53 dist-test-slave-zt01 krb5kdc[4505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:53.110219 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45089
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:45089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:53.213675 4518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:53.213891 4518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:53.213929 4518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:53.217880 4518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:53.217927 4518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:53.217947 4518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:53.217991 4518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:53.218022 4518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:53.222924 4518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:45089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:53.223819 4518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:53.224773 4518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:53.232478 4526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:53.232494 4523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:53.232519 4524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:53.232648 4518 server_base.cc:1034] running on GCE node
I20260620 06:15:53.233249 4518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:53.234217 4518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:53.235385 4518 hybrid_clock.cc:648] HybridClock initialized: now 1781936153235369 us; error 29 us; skew 500 ppm
Jun 20 06:15:53 dist-test-slave-zt01 krb5kdc[4505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:53.237943 4518 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:53.238667 4518 webserver.cc:460] Webserver started at http://127.14.209.190:37977/ using document root <none> and password file <none>
I20260620 06:15:53.239187 4518 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:53.239249 4518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:53.239396 4518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:53.241159 4518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "7af042befa8e44a1a4678632d60bd4e4"
format_stamp: "Formatted at 2026-06-20 06:15:53 on dist-test-slave-zt01"
server_key: "5fd34f3983f0c89de3e8a96a3918c950"
server_key_iv: "71aa9c8777a1480d5e8bd51267ce449c"
server_key_version: "encryptionkey@0"
I20260620 06:15:53.241624 4518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "7af042befa8e44a1a4678632d60bd4e4"
format_stamp: "Formatted at 2026-06-20 06:15:53 on dist-test-slave-zt01"
server_key: "5fd34f3983f0c89de3e8a96a3918c950"
server_key_iv: "71aa9c8777a1480d5e8bd51267ce449c"
server_key_version: "encryptionkey@0"
I20260620 06:15:53.245003 4518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:53.247217 4533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:53.248307 4518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:53.248437 4518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "7af042befa8e44a1a4678632d60bd4e4"
format_stamp: "Formatted at 2026-06-20 06:15:53 on dist-test-slave-zt01"
server_key: "5fd34f3983f0c89de3e8a96a3918c950"
server_key_iv: "71aa9c8777a1480d5e8bd51267ce449c"
server_key_version: "encryptionkey@0"
I20260620 06:15:53.248554 4518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:53.253229 4518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:53.255903 4518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:53.256079 4518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:53.262207 4518 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:45089
I20260620 06:15:53.262223 4585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:45089 every 8 connection(s)
I20260620 06:15:53.262997 4518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:53.265157 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4518
I20260620 06:15:53.265309 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:53.265587 15174 external_mini_cluster.cc:1428] Setting key 75f96513a9dae2b7c9c283401332e37a
I20260620 06:15:53.265666 4586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:53.270607 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:53.266739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45797 (local address 127.14.209.190:45089)
0620 06:15:53.267135 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:53.267154 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:53.267364 (+ 210us) server_negotiation.cc:407] Connection header received
0620 06:15:53.268314 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:53.268330 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:53.268566 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:53.268878 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:53.269612 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:53.269616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:53.269626 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:53.269636 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:53.269683 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:53.269701 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:53.269845 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:53.269910 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 06:15:53.270044 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
I20260620 06:15:53.271374 4586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4: Bootstrap starting.
I20260620 06:15:53.273178 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:53.273705 4586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:53.274474 4586 log.cc:824] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:53.276489 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:53.275085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36067 (local address 127.14.209.190:45089)
0620 06:15:53.275186 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:53.275190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:53.275328 (+ 138us) server_negotiation.cc:407] Connection header received
0620 06:15:53.275430 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:53.275432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:53.275455 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:53.275514 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:53.276139 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:53.276141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:53.276143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:53.276152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:53.276172 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:53.276181 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:53.276248 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:53.276355 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 06:15:53.276380 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:15:53.276612 4586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4: No bootstrap required, opened a new log
I20260620 06:15:53.280329 4586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af042befa8e44a1a4678632d60bd4e4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45089 } }
I20260620 06:15:53.280537 4586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:53.280599 4586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af042befa8e44a1a4678632d60bd4e4, State: Initialized, Role: FOLLOWER
I20260620 06:15:53.280958 4586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af042befa8e44a1a4678632d60bd4e4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45089 } }
I20260620 06:15:53.281100 4586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:53.281183 4586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:53.281275 4586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:53.282172 4586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af042befa8e44a1a4678632d60bd4e4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45089 } }
I20260620 06:15:53.282495 4586 leader_election.cc:302] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af042befa8e44a1a4678632d60bd4e4; no voters:
I20260620 06:15:53.282768 4586 leader_election.cc:288] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:53.282919 4596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:53.283143 4596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7af042befa8e44a1a4678632d60bd4e4, State: Running, Role: LEADER
I20260620 06:15:53.283483 4596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af042befa8e44a1a4678632d60bd4e4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45089 } }
I20260620 06:15:53.283895 4586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:53.285017 4598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af042befa8e44a1a4678632d60bd4e4. Latest consensus state: current_term: 1 leader_uuid: "7af042befa8e44a1a4678632d60bd4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af042befa8e44a1a4678632d60bd4e4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45089 } } }
I20260620 06:15:53.285151 4598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:53.285336 4597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af042befa8e44a1a4678632d60bd4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af042befa8e44a1a4678632d60bd4e4" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45089 } } }
I20260620 06:15:53.285509 4597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:53.285586 4605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:53.288362 4605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:53.293269 4605 catalog_manager.cc:1347] Generated new cluster ID: 5c893e7ca9bf4ae5ac3cec1965e6aed6
I20260620 06:15:53.293368 4605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:53.322360 4605 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:53.322837 4605 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:53.328480 4605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af042befa8e44a1a4678632d60bd4e4: Generated new TSK 0
I20260620 06:15:53.329133 4605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:53.332163 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4518
2026-06-20T06:15:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:53 dist-test-slave-zt01 krb5kdc[4617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:53 dist-test-slave-zt01 krb5kdc[4617](info): set up 2 sockets
Jun 20 06:15:53 dist-test-slave-zt01 krb5kdc[4617](info): commencing operation
krb5kdc: starting...
W20260620 06:15:54.367323 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:54 dist-test-slave-zt01 krb5kdc[4617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:54.480068 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39879
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:39879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:54.582753 4630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:54.583022 4630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:54.583135 4630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:54.586992 4630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:54.587095 4630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:54.587138 4630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:54.587170 4630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:54.587203 4630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:54.592207 4630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:39879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:54.593128 4630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:54.594035 4630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:54.602308 4635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:54.602321 4636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:54.602468 4630 server_base.cc:1034] running on GCE node
W20260620 06:15:54.602301 4638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:54.603029 4630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:54.604086 4630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:54.605266 4630 hybrid_clock.cc:648] HybridClock initialized: now 1781936154605249 us; error 30 us; skew 500 ppm
Jun 20 06:15:54 dist-test-slave-zt01 krb5kdc[4617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:54.607913 4630 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:54.608623 4630 webserver.cc:460] Webserver started at http://127.14.209.190:36981/ using document root <none> and password file <none>
I20260620 06:15:54.609079 4630 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:54.609133 4630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:54.609272 4630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:54.611044 4630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "26481d9d691c468fa85b97dddb15de07"
format_stamp: "Formatted at 2026-06-20 06:15:54 on dist-test-slave-zt01"
server_key: "06abc3f30b6f7abb3f4c0736c98117bd"
server_key_iv: "b9573d84303a5e681a7ec078abb9f6c6"
server_key_version: "encryptionkey@0"
I20260620 06:15:54.611584 4630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "26481d9d691c468fa85b97dddb15de07"
format_stamp: "Formatted at 2026-06-20 06:15:54 on dist-test-slave-zt01"
server_key: "06abc3f30b6f7abb3f4c0736c98117bd"
server_key_iv: "b9573d84303a5e681a7ec078abb9f6c6"
server_key_version: "encryptionkey@0"
I20260620 06:15:54.614810 4630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:54.617009 4645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:54.618141 4630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:54.618228 4630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "26481d9d691c468fa85b97dddb15de07"
format_stamp: "Formatted at 2026-06-20 06:15:54 on dist-test-slave-zt01"
server_key: "06abc3f30b6f7abb3f4c0736c98117bd"
server_key_iv: "b9573d84303a5e681a7ec078abb9f6c6"
server_key_version: "encryptionkey@0"
I20260620 06:15:54.618302 4630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:54.636569 4630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:54.639434 4630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:54.639606 4630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:54.646073 4630 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:39879
I20260620 06:15:54.646098 4697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:39879 every 8 connection(s)
I20260620 06:15:54.646865 4630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:54.649354 4698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:54.654327 4698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07: Bootstrap starting.
I20260620 06:15:54.654904 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4630
I20260620 06:15:54.655040 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:54.655334 15174 external_mini_cluster.cc:1428] Setting key 2c81e9d92145509115662d1ce3ab3d97
I20260620 06:15:54.656865 4698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:54.657544 4698 log.cc:824] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:54.659250 4698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07: No bootstrap required, opened a new log
I20260620 06:15:54.662326 4698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26481d9d691c468fa85b97dddb15de07" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39879 } }
I20260620 06:15:54.662539 4698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:54.662616 4698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26481d9d691c468fa85b97dddb15de07, State: Initialized, Role: FOLLOWER
I20260620 06:15:54.663033 4698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26481d9d691c468fa85b97dddb15de07" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39879 } }
Jun 20 06:15:54 dist-test-slave-zt01 krb5kdc[4617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:54.663230 4698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:54.663300 4698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:54.663391 4698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:54.664296 4698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26481d9d691c468fa85b97dddb15de07" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39879 } }
I20260620 06:15:54.664673 4698 leader_election.cc:302] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26481d9d691c468fa85b97dddb15de07; no voters:
I20260620 06:15:54.665027 4698 leader_election.cc:288] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:54.665093 4703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:54.665284 4703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [term 1 LEADER]: Becoming Leader. State: Replica: 26481d9d691c468fa85b97dddb15de07, State: Running, Role: LEADER
I20260620 06:15:54.665597 4703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26481d9d691c468fa85b97dddb15de07" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39879 } }
I20260620 06:15:54.666160 4698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:54.666723 4705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26481d9d691c468fa85b97dddb15de07. Latest consensus state: current_term: 1 leader_uuid: "26481d9d691c468fa85b97dddb15de07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26481d9d691c468fa85b97dddb15de07" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39879 } } }
I20260620 06:15:54.666834 4705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:54.667224 4704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26481d9d691c468fa85b97dddb15de07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26481d9d691c468fa85b97dddb15de07" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 39879 } } }
I20260620 06:15:54.667328 4704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:54.669193 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:54.656384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60555 (local address 127.14.209.190:39879)
0620 06:15:54.656761 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:54.656774 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:54.656805 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:15:54.657421 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:54.657436 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:54.657769 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:54.658069 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:54.658814 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:54.659827 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:54.660674 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:54.661111 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:54.661272 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:54.663748 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:54.663754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:54.663773 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:54.663796 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:54.665737 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:54.666192 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:54.666198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:54.666203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:54.666331 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:54.666656 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:54.666660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:54.666662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:54.667174 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:54.667354 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:54.668124 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 06:15:54.668381 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260620 06:15:54.671486 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 06:15:54.671831 4718 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:54.671905 4718 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:54.671965 4709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:54.672748 4709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:54.678117 4709 catalog_manager.cc:1347] Generated new cluster ID: 8ec33d33af9d47b4a51b248cf086afd3
I20260620 06:15:54.678223 4709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:54.681250 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:54.673387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44033 (local address 127.14.209.190:39879)
0620 06:15:54.673482 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:54.673487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:54.673687 (+ 200us) server_negotiation.cc:407] Connection header received
0620 06:15:54.673805 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:54.673807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:54.673843 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:54.673932 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:54.674561 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:54.675280 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:54.676344 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:54.676647 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:54.676705 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:54.678298 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:54.678301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:54.678304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:54.678325 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:54.679695 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:54.680100 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:54.680105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:54.680106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:54.680181 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:54.680446 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:54.680450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:54.680452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:54.680608 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:54.680750 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:54.680978 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 06:15:54.681082 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:15:54.698173 4709 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:54.699332 4709 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:54.708976 4709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26481d9d691c468fa85b97dddb15de07: Generated new TSK 0
I20260620 06:15:54.709687 4709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:54.721326 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4630
2026-06-20T06:15:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:54 dist-test-slave-zt01 krb5kdc[4729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:54 dist-test-slave-zt01 krb5kdc[4729](info): set up 2 sockets
Jun 20 06:15:54 dist-test-slave-zt01 krb5kdc[4729](info): commencing operation
krb5kdc: starting...
W20260620 06:15:55.755226 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:55 dist-test-slave-zt01 krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:55.867977 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33353
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:33353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:55.971029 4742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:55.971266 4742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:55.971305 4742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:55.975131 4742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:55.975175 4742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:55.975194 4742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:55.975211 4742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:55.975225 4742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:55.980079 4742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:33353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:33353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:55.980898 4742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:55.981742 4742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:55.989596 4748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:55.989605 4750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:55.989651 4742 server_base.cc:1034] running on GCE node
W20260620 06:15:55.989605 4747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:55.990271 4742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:55.991293 4742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:55.992486 4742 hybrid_clock.cc:648] HybridClock initialized: now 1781936155992460 us; error 38 us; skew 500 ppm
Jun 20 06:15:55 dist-test-slave-zt01 krb5kdc[4729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936155, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:55.998980 4742 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:55.999825 4742 webserver.cc:460] Webserver started at http://127.14.209.190:38963/ using document root <none> and password file <none>
I20260620 06:15:56.000344 4742 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:56.000433 4742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:56.000654 4742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:56.002548 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "1981e73ee4de4c47ba3d6f13f62afe44"
format_stamp: "Formatted at 2026-06-20 06:15:55 on dist-test-slave-zt01"
server_key: "f7c0af5e52a9174c834490ef3544c2e2"
server_key_iv: "7ec83046d8fa48e0634a57d225040bc5"
server_key_version: "encryptionkey@0"
I20260620 06:15:56.003175 4742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "1981e73ee4de4c47ba3d6f13f62afe44"
format_stamp: "Formatted at 2026-06-20 06:15:55 on dist-test-slave-zt01"
server_key: "f7c0af5e52a9174c834490ef3544c2e2"
server_key_iv: "7ec83046d8fa48e0634a57d225040bc5"
server_key_version: "encryptionkey@0"
I20260620 06:15:56.006619 4742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:56.008898 4757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:56.010011 4742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:56.010151 4742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "1981e73ee4de4c47ba3d6f13f62afe44"
format_stamp: "Formatted at 2026-06-20 06:15:55 on dist-test-slave-zt01"
server_key: "f7c0af5e52a9174c834490ef3544c2e2"
server_key_iv: "7ec83046d8fa48e0634a57d225040bc5"
server_key_version: "encryptionkey@0"
I20260620 06:15:56.010253 4742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:56.036793 4742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:56.039635 4742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:56.039829 4742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:56.046492 4742 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:33353
I20260620 06:15:56.046514 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:33353 every 8 connection(s)
I20260620 06:15:56.047408 4742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:56.050011 4810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:56.053162 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4742
I20260620 06:15:56.053272 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:56.053593 15174 external_mini_cluster.cc:1428] Setting key ddea857478833d66a96ebac51f6ee8c8
I20260620 06:15:56.056137 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44: Bootstrap starting.
I20260620 06:15:56.058943 4810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:56.059727 4810 log.cc:824] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:56.061806 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44: No bootstrap required, opened a new log
I20260620 06:15:56.063542 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:56.054908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60161 (local address 127.14.209.190:33353)
0620 06:15:56.055356 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:56.055376 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:56.055429 (+ 53us) server_negotiation.cc:407] Connection header received
0620 06:15:56.056279 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:56.056301 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:56.056719 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:56.057072 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:56.057518 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:56.059151 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:56.060182 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:56.060766 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:56.060922 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:56.061711 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:56.061716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:56.061731 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:56.062196 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:56.062254 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:56.062293 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:56.062390 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:56.062498 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 06:15:56.062677 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":155,"threads_started":1}
I20260620 06:15:56.065904 4810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33353 } }
I20260620 06:15:56.066097 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:56.066179 4810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:56.066258 4810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1981e73ee4de4c47ba3d6f13f62afe44, State: Initialized, Role: FOLLOWER
I20260620 06:15:56.066622 4810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33353 } }
I20260620 06:15:56.066761 4810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:56.066850 4810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:56.066946 4810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:56.067849 4810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33353 } }
I20260620 06:15:56.068193 4810 leader_election.cc:302] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1981e73ee4de4c47ba3d6f13f62afe44; no voters:
I20260620 06:15:56.068518 4810 leader_election.cc:288] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:56.068617 4820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:56.068861 4820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [term 1 LEADER]: Becoming Leader. State: Replica: 1981e73ee4de4c47ba3d6f13f62afe44, State: Running, Role: LEADER
I20260620 06:15:56.069213 4820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33353 } }
I20260620 06:15:56.069667 4810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:56.070240 4820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1981e73ee4de4c47ba3d6f13f62afe44. Latest consensus state: current_term: 1 leader_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33353 } } }
I20260620 06:15:56.070437 4820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:56.070652 4822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1981e73ee4de4c47ba3d6f13f62afe44" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 33353 } } }
I20260620 06:15:56.070755 4822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:56.070871 4829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:56.072860 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:56.068089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49545 (local address 127.14.209.190:33353)
0620 06:15:56.068207 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:56.068211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:56.068390 (+ 179us) server_negotiation.cc:407] Connection header received
0620 06:15:56.068570 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:56.068573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:56.068621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:56.068709 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:56.069001 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:56.069701 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:56.070893 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:56.071306 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:56.071405 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:56.072409 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:56.072413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:56.072415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:56.072427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:56.072454 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:56.072467 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:56.072517 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:56.072683 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 06:15:56.072733 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:15:56.073803 4829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:56.078477 4829 catalog_manager.cc:1347] Generated new cluster ID: 0fd1c638fb75448bbd313471aa5d6482
I20260620 06:15:56.078531 4829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:56.086583 4829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:56.087553 4829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:56.092592 4829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1981e73ee4de4c47ba3d6f13f62afe44: Generated new TSK 0
I20260620 06:15:56.093114 4829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:56.096122 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4742
2026-06-20T06:15:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:56 dist-test-slave-zt01 krb5kdc[4841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:56 dist-test-slave-zt01 krb5kdc[4841](info): set up 2 sockets
Jun 20 06:15:56 dist-test-slave-zt01 krb5kdc[4841](info): commencing operation
krb5kdc: starting...
W20260620 06:15:57.127382 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:57 dist-test-slave-zt01 krb5kdc[4841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:57.242074 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45331
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:57.358698 4854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:57.358968 4854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:57.359094 4854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:57.363039 4854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:57.363142 4854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:57.363186 4854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:57.363224 4854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:57.363253 4854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:57.368446 4854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:45331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:57.369385 4854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:57.370285 4854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:15:57.378393 4854 server_base.cc:1034] running on GCE node
W20260620 06:15:57.378329 4859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:57.378329 4862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:57.378389 4860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:57.379163 4854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:57.380111 4854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:57.381263 4854 hybrid_clock.cc:648] HybridClock initialized: now 1781936157381248 us; error 30 us; skew 500 ppm
Jun 20 06:15:57 dist-test-slave-zt01 krb5kdc[4841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936157, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:57.383764 4854 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:57.384572 4854 webserver.cc:460] Webserver started at http://127.14.209.190:33773/ using document root <none> and password file <none>
I20260620 06:15:57.385075 4854 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:57.385131 4854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:57.385363 4854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:57.387208 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "e23034fffb06447eaa48c1f1b4037821"
format_stamp: "Formatted at 2026-06-20 06:15:57 on dist-test-slave-zt01"
server_key: "01ee2e39e2c240c384ef6131189b34ff"
server_key_iv: "e4254d44119b9f776a031ab7298ec669"
server_key_version: "encryptionkey@0"
I20260620 06:15:57.387723 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "e23034fffb06447eaa48c1f1b4037821"
format_stamp: "Formatted at 2026-06-20 06:15:57 on dist-test-slave-zt01"
server_key: "01ee2e39e2c240c384ef6131189b34ff"
server_key_iv: "e4254d44119b9f776a031ab7298ec669"
server_key_version: "encryptionkey@0"
I20260620 06:15:57.390976 4854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:57.393110 4869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:57.394188 4854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:57.394320 4854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "e23034fffb06447eaa48c1f1b4037821"
format_stamp: "Formatted at 2026-06-20 06:15:57 on dist-test-slave-zt01"
server_key: "01ee2e39e2c240c384ef6131189b34ff"
server_key_iv: "e4254d44119b9f776a031ab7298ec669"
server_key_version: "encryptionkey@0"
I20260620 06:15:57.394423 4854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:57.398708 4854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:57.401283 4854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:57.401460 4854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:57.408109 4854 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:45331
I20260620 06:15:57.408138 4921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:45331 every 8 connection(s)
I20260620 06:15:57.408919 4854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:57.411381 4922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:57.416337 4922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821: Bootstrap starting.
I20260620 06:15:57.417074 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4854
I20260620 06:15:57.417197 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:57.417474 15174 external_mini_cluster.cc:1428] Setting key 2bc40413c8e86ae9aec54b1b32b11ed5
I20260620 06:15:57.418807 4922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:57.419554 4922 log.cc:824] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:57.421401 4922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821: No bootstrap required, opened a new log
I20260620 06:15:57.421955 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:57.418427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39887 (local address 127.14.209.190:45331)
0620 06:15:57.418812 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:57.418824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:57.418855 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:15:57.419516 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:57.419532 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:57.419820 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:57.420122 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:57.420833 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:57.420838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:57.420848 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:57.420859 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:57.420906 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:57.420923 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:57.421034 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:57.421141 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 06:15:57.421289 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260620 06:15:57.424467 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:57.424997 4922 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23034fffb06447eaa48c1f1b4037821" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45331 } }
I20260620 06:15:57.425211 4922 raft_consensus.cc:374] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:15:57.425295 4922 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23034fffb06447eaa48c1f1b4037821, State: Initialized, Role: FOLLOWER
I20260620 06:15:57.425729 4922 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23034fffb06447eaa48c1f1b4037821" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45331 } }
I20260620 06:15:57.425874 4922 raft_consensus.cc:388] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:15:57.425941 4922 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:15:57.426057 4922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:57.426964 4922 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23034fffb06447eaa48c1f1b4037821" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45331 } }
I20260620 06:15:57.427342 4922 leader_election.cc:302] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e23034fffb06447eaa48c1f1b4037821; no voters:
I20260620 06:15:57.427660 4922 leader_election.cc:288] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:15:57.427778 4932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:57.427996 4932 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [term 1 LEADER]: Becoming Leader. State: Replica: e23034fffb06447eaa48c1f1b4037821, State: Running, Role: LEADER
I20260620 06:15:57.428215 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:57.426518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36251 (local address 127.14.209.190:45331)
0620 06:15:57.426635 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:57.426642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:57.426826 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:15:57.426937 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:57.426940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:57.426968 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:57.427060 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:57.427758 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:57.427762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:57.427764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:15:57.427777 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:57.427803 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:15:57.427819 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:57.427879 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:57.428079 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 06:15:57.428117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:15:57.428436 4932 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23034fffb06447eaa48c1f1b4037821" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45331 } }
I20260620 06:15:57.428859 4922 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:57.430270 4934 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23034fffb06447eaa48c1f1b4037821. Latest consensus state: current_term: 1 leader_uuid: "e23034fffb06447eaa48c1f1b4037821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23034fffb06447eaa48c1f1b4037821" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45331 } } }
I20260620 06:15:57.430398 4934 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:57.430768 4943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:57.430719 4933 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e23034fffb06447eaa48c1f1b4037821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23034fffb06447eaa48c1f1b4037821" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 45331 } } }
I20260620 06:15:57.430820 4933 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:57.433230 4943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:57.438230 4943 catalog_manager.cc:1347] Generated new cluster ID: fdeff19eb97a4644a518f795c4bac128
I20260620 06:15:57.438295 4943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:57.453176 4943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:57.453466 4943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:57.459607 4943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e23034fffb06447eaa48c1f1b4037821: Generated new TSK 0
I20260620 06:15:57.460167 4943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:15:57.468366 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4854
2026-06-20T06:15:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1372 ms)
[----------] 18 tests from AuthTokenIssuingTest (25001 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:15:57 dist-test-slave-zt01 krb5kdc[4953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:15:57 dist-test-slave-zt01 krb5kdc[4953](info): set up 2 sockets
Jun 20 06:15:57 dist-test-slave-zt01 krb5kdc[4953](info): commencing operation
krb5kdc: starting...
W20260620 06:15:58.499262 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:15:58 dist-test-slave-zt01 krb5kdc[4953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:15:58.610589 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:42505
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:42505,127.14.209.189:38947,127.14.209.188:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:15:58.714888 4966 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:15:58.715235 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:58.715319 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:58.715404 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:58.719291 4966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:58.719374 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:58.719419 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:58.719455 4966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:58.719491 4966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:58.725106 4966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:42505,127.14.209.189:38947,127.14.209.188:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:58.726022 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:58.726927 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:58.734884 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:58.734879 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:58.734884 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:58.734939 4966 server_base.cc:1034] running on GCE node
I20260620 06:15:58.735720 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:58.736791 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:58.737998 4966 hybrid_clock.cc:648] HybridClock initialized: now 1781936158737972 us; error 39 us; skew 500 ppm
Jun 20 06:15:58 dist-test-slave-zt01 krb5kdc[4953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:58.740883 4966 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:15:58.741711 4966 webserver.cc:460] Webserver started at http://127.14.209.190:35313/ using document root <none> and password file <none>
I20260620 06:15:58.742269 4966 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:58.742367 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:58.742602 4966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:58.744719 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "26e9e11ba67a43f9b5e3f801d31ebe98"
format_stamp: "Formatted at 2026-06-20 06:15:58 on dist-test-slave-zt01"
server_key: "1c85388b4fa7a255e8969ba803402ec5"
server_key_iv: "035a0b80b7f3639cfdf8bf9284ea483e"
server_key_version: "encryptionkey@0"
I20260620 06:15:58.745337 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "26e9e11ba67a43f9b5e3f801d31ebe98"
format_stamp: "Formatted at 2026-06-20 06:15:58 on dist-test-slave-zt01"
server_key: "1c85388b4fa7a255e8969ba803402ec5"
server_key_iv: "035a0b80b7f3639cfdf8bf9284ea483e"
server_key_version: "encryptionkey@0"
I20260620 06:15:58.748965 4966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:15:58.751327 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:58.752552 4966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:15:58.752799 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "26e9e11ba67a43f9b5e3f801d31ebe98"
format_stamp: "Formatted at 2026-06-20 06:15:58 on dist-test-slave-zt01"
server_key: "1c85388b4fa7a255e8969ba803402ec5"
server_key_iv: "035a0b80b7f3639cfdf8bf9284ea483e"
server_key_version: "encryptionkey@0"
I20260620 06:15:58.752919 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:58.769600 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:58.772454 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:58.772657 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:58.779739 4966 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:42505
I20260620 06:15:58.780359 5033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:42505 every 8 connection(s)
I20260620 06:15:58.780707 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:15:58.783206 5034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:58.785789 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 4966
I20260620 06:15:58.785948 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:15:58.786228 15174 external_mini_cluster.cc:1428] Setting key 36af12a1658d887fc2bcb182296a04ef
I20260620 06:15:58.786890 5034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.209.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.189@KRBTEST.COM" created.
Jun 20 06:15:58 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:15:58.802979 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.789285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42505 (local address 127.0.0.1:36148)
0620 06:15:58.789663 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.789699 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:58.790522 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:58.791153 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:58.791157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:58.791558 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:58.791765 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:58.791777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:58.792889 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:58.792893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:58.793585 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:58.793594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:58.794177 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:58.794180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:58.794347 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:58.794927 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:58.794942 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:58.798106 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:58.800384 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:58.800391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:58.800397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:58.800628 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:58.801001 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:58.801005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:58.801007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:58.801122 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:58.801816 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:58.801820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:58.801927 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:15:58.802052 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:15:58.802253 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":44,"thread_start_us":98,"threads_started":1}
I20260620 06:15:58.802986 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.789921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.14.209.190:42505)
0620 06:15:58.790167 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:58.790180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:58.790218 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:15:58.790794 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:58.790798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:58.790956 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:58.791156 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:58.791906 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:58.792766 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:58.793701 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:58.794039 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:58.794236 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:58.798255 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:58.798264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:58.798279 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:58.798309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:58.800272 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:58.800739 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:58.800748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:58.800753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:58.800889 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:58.801219 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:58.801225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:58.801228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:58.801700 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:58.801846 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:58.802171 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 06:15:58.802331 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260620 06:15:58.805354 5034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:58.806025 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.805792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41360)
0620 06:15:58.805897 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.805977 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:15:58.806252 4984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
I20260620 06:15:58.807111 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.806861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41368)
0620 06:15:58.806963 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.807010 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:15:58.807279 5034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:38947: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab.
Entry for principal kudu/127.14.209.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab.
I20260620 06:15:58.814200 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:38947
--webserver_interface=127.14.209.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab
--principal=kudu/127.14.209.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:42505,127.14.209.189:38947,127.14.209.188:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:15:58.856505 5034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } attempt: 1
I20260620 06:15:58.857313 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.857042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41370)
0620 06:15:58.857181 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.857245 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 06:15:58.858000 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.857809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41382)
0620 06:15:58.857907 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.857951 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:15:58.858193 5034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:38947: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
W20260620 06:15:58.917469 5041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:15:58.917711 5041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:58.917733 5041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:58.917912 5041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:58.921768 5041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:58.921821 5041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:58.921840 5041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:58.921856 5041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:58.921906 5041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:58.926383 5034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } attempt: 2
I20260620 06:15:58.927220 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.926916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41392)
0620 06:15:58.927089 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.927148 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 06:15:58.927517 5041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:42505,127.14.209.189:38947,127.14.209.188:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab
--principal=kudu/127.14.209.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.209.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:58.927956 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.927746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41400)
0620 06:15:58.927836 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.927886 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 06:15:58.928081 5034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:38947: Network error: Client connection negotiation failed: client connection to 127.14.209.189:38947: connect: Connection refused (error 111)
I20260620 06:15:58.928609 5041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:58.929548 5041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:15:58.937624 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:58.937641 5047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:58.937664 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:58.937675 5041 server_base.cc:1034] running on GCE node
I20260620 06:15:58.938344 5041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:58.939407 5041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:58.940598 5041 hybrid_clock.cc:648] HybridClock initialized: now 1781936158940582 us; error 27 us; skew 500 ppm
Jun 20 06:15:58 dist-test-slave-zt01 krb5kdc[4953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:58.943271 5041 init.cc:377] Logged in from keytab as kudu/127.14.209.189@KRBTEST.COM (short username kudu)
I20260620 06:15:58.944101 5041 webserver.cc:460] Webserver started at http://127.14.209.189:45831/ using document root <none> and password file <none>
I20260620 06:15:58.944566 5041 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:58.944619 5041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:58.944764 5041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:58.946558 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data/instance:
uuid: "0040927908894e02a415835d2e1fea0d"
format_stamp: "Formatted at 2026-06-20 06:15:58 on dist-test-slave-zt01"
server_key: "e5fb819a48267e9bb912c55369df190f"
server_key_iv: "669f7612a051815d5a24cf3b26d58499"
server_key_version: "encryptionkey@0"
I20260620 06:15:58.947036 5041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal/instance:
uuid: "0040927908894e02a415835d2e1fea0d"
format_stamp: "Formatted at 2026-06-20 06:15:58 on dist-test-slave-zt01"
server_key: "e5fb819a48267e9bb912c55369df190f"
server_key_iv: "669f7612a051815d5a24cf3b26d58499"
server_key_version: "encryptionkey@0"
I20260620 06:15:58.950320 5041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:15:58.952538 5056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:58.953667 5041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:15:58.953781 5041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal
uuid: "0040927908894e02a415835d2e1fea0d"
format_stamp: "Formatted at 2026-06-20 06:15:58 on dist-test-slave-zt01"
server_key: "e5fb819a48267e9bb912c55369df190f"
server_key_iv: "669f7612a051815d5a24cf3b26d58499"
server_key_version: "encryptionkey@0"
I20260620 06:15:58.953879 5041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:58.979884 5041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:58.982968 5041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:58.983170 5041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:58.989686 5041 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.189:38947
I20260620 06:15:58.989984 5108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.189:38947 every 8 connection(s)
I20260620 06:15:58.990588 5041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
I20260620 06:15:58.993096 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:58.996541 5109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:58.999635 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 5041
I20260620 06:15:58.999773 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-1/wal/instance
I20260620 06:15:59.000090 15174 external_mini_cluster.cc:1428] Setting key cfd1abb0620c54b19338ef7943f53325
Jun 20 06:15:58 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.188@KRBTEST.COM" created.
I20260620 06:15:59.012135 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.998227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.14.209.190:42505)
0620 06:15:58.998385 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:58.998389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:58.999090 (+ 701us) server_negotiation.cc:407] Connection header received
0620 06:15:59.000251 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.000254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.000301 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.000373 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.001887 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.002765 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.004007 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.004438 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.004558 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.007177 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.007184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.007189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.007218 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.009434 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.009959 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.009962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.009964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.010046 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.010582 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.010588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.010589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.011173 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.011307 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.011900 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.012018 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:15:59.013070 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:58.998521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42505 (local address 127.0.0.1:36160)
0620 06:15:58.998927 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:58.998968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.000032 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.000573 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.000584 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.001053 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.001706 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.001729 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.002943 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.002946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.003876 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.003887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.004600 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.004603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.004756 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.005390 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.005406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.006985 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.009545 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.009557 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.009568 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.009815 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.010192 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.010197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.010202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.010347 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.011318 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.011323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.011604 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.011812 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.012149 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260620 06:15:59.014492 5109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:59.019227 5034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } attempt: 3
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
I20260620 06:15:59.024889 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.015013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41416)
0620 06:15:59.015159 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.015171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.015239 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.015831 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.015833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.016001 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.016065 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.016070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.016987 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.016990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.017490 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.017497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.018036 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.018039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.018124 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.018551 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.018570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.021311 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.023429 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.023435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.023438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.023620 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.023909 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.023914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.023916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.023994 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.024419 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.024422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.024504 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.024601 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.024715 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 06:15:59.025220 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.015201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.14.209.189:38947)
0620 06:15:59.015504 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.015512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.015528 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:15:59.015557 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.015561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.015683 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.015851 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.016168 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.016869 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.017595 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.017929 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.018041 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.021432 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.021444 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.021458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.021484 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.023330 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.023724 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.023731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.023733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.023835 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.024084 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.024087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.024089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.024312 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.024450 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.024710 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.024858 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab.
Entry for principal kudu/127.14.209.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab.
I20260620 06:15:59.026625 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:36013
--webserver_interface=127.14.209.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab
--principal=kudu/127.14.209.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:39381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:42505,127.14.209.189:38947,127.14.209.188:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:15:59.026957 5109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:59.027694 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.027427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35962)
0620 06:15:59.027557 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.027634 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 06:15:59.027921 5059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
I20260620 06:15:59.028764 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.028512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35964)
0620 06:15:59.028642 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.028699 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 06:15:59.028952 5109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:36013: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
I20260620 06:15:59.029320 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.019597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41418)
0620 06:15:59.019726 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.019738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.019863 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.020165 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.020168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.020465 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.020559 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.020565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.021480 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.021482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.022284 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.022295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.022943 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.022946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.023015 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.023471 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.023491 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.024754 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.027806 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.027812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.027817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.028076 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.028389 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.028394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.028395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.028484 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.028915 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.028919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.029008 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.029103 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.029216 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:15:59.029805 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.019650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.14.209.189:38947)
0620 06:15:59.019970 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.019979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.019990 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:15:59.020026 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.020028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.020057 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.020151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.020679 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.021374 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.022414 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.022818 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.022924 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.024890 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.024894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.024900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.024928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.027668 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.028190 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.028195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.028197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.028278 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.028599 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.028603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.028605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.028780 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.029504 (+ 724us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.029580 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.029678 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260620 06:15:59.030402 5034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:59.030999 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.030789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35966)
0620 06:15:59.030888 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.030936 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 06:15:59.031620 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.031403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35982)
0620 06:15:59.031527 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.031569 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 06:15:59.031772 5034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:36013: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
I20260620 06:15:59.062920 5034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } attempt: 1
I20260620 06:15:59.063539 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.063320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35984)
0620 06:15:59.063437 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.063489 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 06:15:59.064196 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.064009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35988)
0620 06:15:59.064103 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.064147 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:15:59.064343 5034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:36013: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
I20260620 06:15:59.078213 5109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } attempt: 1
I20260620 06:15:59.078895 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.078604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:35994)
0620 06:15:59.078752 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.078821 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 06:15:59.079615 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.079396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36002)
0620 06:15:59.079506 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.079557 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:15:59.079802 5109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:36013: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
W20260620 06:15:59.131542 5116 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:15:59.131764 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:15:59.131786 5116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:15:59.131821 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:15:59.135651 5116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:15:59.135715 5116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:15:59.135735 5116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:15:59.135751 5116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:15:59.135766 5116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:15:59.139492 5034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } attempt: 2
I20260620 06:15:59.140422 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.140087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36018)
0620 06:15:59.140273 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.140337 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 06:15:59.142030 5116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:39381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:42505,127.14.209.189:38947,127.14.209.188:36013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab
--principal=kudu/127.14.209.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.209.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:15:59.143317 5116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:15:59.143553 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.143230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36032)
0620 06:15:59.143404 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.143470 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 06:15:59.143718 5034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:36013: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
I20260620 06:15:59.144629 5116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:15:59.147943 5109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } attempt: 2
I20260620 06:15:59.148593 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.148327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36048)
0620 06:15:59.148461 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.148519 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:15:59.149322 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.149057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36050)
0620 06:15:59.149186 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.149240 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:15:59.149466 5109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:36013: Network error: Client connection negotiation failed: client connection to 127.14.209.188:36013: connect: Connection refused (error 111)
W20260620 06:15:59.153762 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:59.153811 5116 server_base.cc:1034] running on GCE node
W20260620 06:15:59.153803 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:15:59.153923 5122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:15:59.154562 5116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:15:59.155586 5116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:15:59.156780 5116 hybrid_clock.cc:648] HybridClock initialized: now 1781936159156748 us; error 40 us; skew 500 ppm
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936159, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:15:59.159514 5116 init.cc:377] Logged in from keytab as kudu/127.14.209.188@KRBTEST.COM (short username kudu)
I20260620 06:15:59.160367 5116 webserver.cc:460] Webserver started at http://127.14.209.188:36785/ using document root <none> and password file <none>
I20260620 06:15:59.160904 5116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:15:59.160996 5116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:15:59.161190 5116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:15:59.163204 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data/instance:
uuid: "280c432c047e4ca7ad1658f86f7ef14d"
format_stamp: "Formatted at 2026-06-20 06:15:59 on dist-test-slave-zt01"
server_key: "7c7cdd7c61a5d0436a1b8fcd52c74c22"
server_key_iv: "be0cb130012c019c164ce8bfab8dbfdd"
server_key_version: "encryptionkey@0"
I20260620 06:15:59.163764 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal/instance:
uuid: "280c432c047e4ca7ad1658f86f7ef14d"
format_stamp: "Formatted at 2026-06-20 06:15:59 on dist-test-slave-zt01"
server_key: "7c7cdd7c61a5d0436a1b8fcd52c74c22"
server_key_iv: "be0cb130012c019c164ce8bfab8dbfdd"
server_key_version: "encryptionkey@0"
I20260620 06:15:59.167254 5116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:15:59.169467 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:15:59.170579 5116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:15:59.170712 5116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal
uuid: "280c432c047e4ca7ad1658f86f7ef14d"
format_stamp: "Formatted at 2026-06-20 06:15:59 on dist-test-slave-zt01"
server_key: "7c7cdd7c61a5d0436a1b8fcd52c74c22"
server_key_iv: "be0cb130012c019c164ce8bfab8dbfdd"
server_key_version: "encryptionkey@0"
I20260620 06:15:59.170835 5116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:15:59.206000 5116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:15:59.208833 5116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:15:59.209045 5116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:15:59.216356 5116 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.188:36013
I20260620 06:15:59.216610 5184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.188:36013 every 8 connection(s)
I20260620 06:15:59.217396 5116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
I20260620 06:15:59.219977 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:15:59.222813 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 5116
I20260620 06:15:59.222939 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935885709390-15174-0/minicluster-data/master-2/wal/instance
I20260620 06:15:59.223238 15174 external_mini_cluster.cc:1428] Setting key 5656f7564b8ffa694031a5e778ed6608
I20260620 06:15:59.223737 5185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936159, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:15:59.235374 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.225446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.14.209.190:42505)
0620 06:15:59.225577 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.225582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.225784 (+ 202us) server_negotiation.cc:407] Connection header received
0620 06:15:59.226111 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.226114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.226165 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.226260 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.227003 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.227544 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.228509 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.228750 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.228878 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.231445 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.231449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.231451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.231475 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.233234 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.233804 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.233810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.233815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.233901 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.234278 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.234281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.234282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.234497 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.234598 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.235050 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.235205 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:15:59.236080 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.225338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:42505 (local address 127.0.0.1:36172)
0620 06:15:59.225662 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.225697 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.225947 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.226311 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.226316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.226639 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.226868 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.226880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.227656 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.227662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.228386 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.228394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.228907 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.228911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.229074 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.229637 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.229663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.231309 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.233344 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.233351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.233358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.233701 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.233996 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.234000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.234002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.234175 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.234599 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.234605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.234817 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.234983 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.235322 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260620 06:15:59.236085 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.224455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.14.209.188:36013)
0620 06:15:59.224864 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.224884 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.224915 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:15:59.225765 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.225784 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.226034 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.226213 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.226989 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.227869 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.228725 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.229084 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.229201 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.231941 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.231949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.231960 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.231984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.233721 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.234186 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.234190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.234191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.234295 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.234615 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.234619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.234621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.234852 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.235047 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.235273 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.235401 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":132,"threads_started":1}
I20260620 06:15:59.237447 5185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:59.240671 5109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } attempt: 3
I20260620 06:15:59.242852 5034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } attempt: 3
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936159, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:15:59.248001 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.238299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.14.209.190:42505)
0620 06:15:59.238393 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.238398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.238428 (+ 30us) server_negotiation.cc:407] Connection header received
0620 06:15:59.238559 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.238561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.238598 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.238670 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.239349 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.240002 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.241031 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.241340 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.241455 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.244212 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.244216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.244219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.244246 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.245988 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.246496 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.246499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.246501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.246584 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.246901 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.246907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.246909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.247152 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.247353 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.247729 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.247845 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:15:59.248733 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.238180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.14.209.189:38947)
0620 06:15:59.238297 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.238301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.238315 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:15:59.238360 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.238363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.238411 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.238535 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.238983 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.239695 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.240706 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.241059 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.241158 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.243459 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.243463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.243465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.243489 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.245975 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.246600 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.246605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.246610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.246692 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.247293 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.247533 (+ 240us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.247535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.247749 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.247832 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.248449 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.248580 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:15:59.249442 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.237917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:38947 (local address 127.0.0.1:41424)
0620 06:15:59.238023 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.238038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.238112 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.238534 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.238540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.238718 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.238858 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.238868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.239833 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.239836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.240560 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.240570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.241284 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.241289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.241370 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.241783 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.241797 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.243295 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.246225 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.246228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.246230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.246495 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.247094 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.247096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.247099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.247184 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.247905 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.247908 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.248250 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.249158 (+ 908us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.249304 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 06:15:59.250101 5185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:15:59.250537 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.241046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36066)
0620 06:15:59.241156 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.241169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.241235 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.241781 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.241784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.241968 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.242051 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.242058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.242854 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.242858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.243614 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.243624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.244306 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.244309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.244408 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.244981 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.245000 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.246380 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.248432 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.248435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.248437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.249054 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.249413 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.249419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.249421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.249510 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.250061 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.250064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.250148 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.250228 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.250348 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 06:15:59.251369 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.241249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.14.209.188:36013)
0620 06:15:59.241590 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.241595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.241607 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:15:59.241642 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.241645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.241687 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.241760 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.242186 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.242708 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.243812 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.244180 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.244589 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.246517 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.246520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.246523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.246544 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.248296 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.249158 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.249161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.249163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.249243 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.249743 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.249747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.249748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.249955 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.251031 (+ 1076us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.251124 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.251237 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260620 06:15:59.253594 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.243206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36070)
0620 06:15:59.243413 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.243425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.243488 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.244097 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.244100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.244285 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.244371 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.244377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.245912 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.245915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.246723 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.246737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.247704 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.247708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.247802 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.248214 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.248231 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.249889 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.252061 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.252067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.252071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.252327 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.252655 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.252661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.252666 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.252752 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.253152 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.253155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.253235 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.253324 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.253446 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 06:15:59.254169 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.243601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.14.209.188:36013)
0620 06:15:59.243891 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.243899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.243909 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:15:59.243935 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.243937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.243969 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.244477 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.244574 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.245756 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.246927 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.247560 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.248007 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.250012 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.250015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.250018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.250041 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.251940 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.252447 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.252450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.252451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.252548 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.252873 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.252875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.252877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.253046 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.253849 (+ 803us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.253920 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.254020 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260620 06:15:59.254750 5109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: Bootstrap starting.
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
Jun 20 06:15:59 dist-test-slave-zt01 krb5kdc[4953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936159, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:15:59.257575 5109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:59.258466 5109 log.cc:824] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:59.260044 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.249461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.14.209.189:38947)
0620 06:15:59.250061 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.250065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.250075 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:15:59.250101 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.250103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.250135 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.250201 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.251267 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.251957 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.253032 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.253345 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.253414 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.256231 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.256234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.256237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.256267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.258088 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.258588 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.258591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.258593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.258671 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.258993 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.258996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.258998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.259234 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.259406 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.259792 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.259891 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
I20260620 06:15:59.260627 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.250829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:36013 (local address 127.0.0.1:36082)
0620 06:15:59.251507 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 06:15:59.251520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:15:59.251588 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:15:59.252076 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:15:59.252079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:15:59.252231 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:15:59.252300 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.252306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.253187 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.253190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.254480 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:15:59.254493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.255041 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.255046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:15:59.255180 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.255626 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:15:59.255640 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:15:59.257043 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:15:59.258991 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.258993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.258995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.259244 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.259508 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:15:59.259510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:15:59.259512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:15:59.259587 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:15:59.259990 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:15:59.259993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:15:59.260065 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 06:15:59.260370 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 06:15:59.260484 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618}
I20260620 06:15:59.260708 5109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: No bootstrap required, opened a new log
I20260620 06:15:59.261279 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.251317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.14.209.188:36013)
0620 06:15:59.251835 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.251839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.251849 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:15:59.251874 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.251877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.251908 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.251977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.252413 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.253097 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.254612 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.254919 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.255018 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.257171 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.257175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.257177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.257197 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.258893 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.259339 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.259342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.259343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.259417 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.259679 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.259683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.259685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.259875 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.259959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.261035 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.261131 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":82,"threads_started":1}
I20260620 06:15:59.261565 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: Bootstrap starting.
I20260620 06:15:59.263422 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:15:59.263975 5034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:59.264151 5185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: Bootstrap starting.
I20260620 06:15:59.264791 5034 log.cc:824] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:59.265547 5109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
I20260620 06:15:59.265765 5109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0040927908894e02a415835d2e1fea0d, State: Initialized, Role: FOLLOWER
I20260620 06:15:59.264705 5008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36176
I20260620 06:15:59.266074 5109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
I20260620 06:15:59.266347 5185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: Neither blocks nor log segments found. Creating new log.
I20260620 06:15:59.266625 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: No bootstrap required, opened a new log
I20260620 06:15:59.266625 5194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.266834 5194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.267132 5185 log.cc:824] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:15:59.267338 5109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:59.268702 5185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: No bootstrap required, opened a new log
I20260620 06:15:59.269572 5034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
I20260620 06:15:59.269977 5034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e9e11ba67a43f9b5e3f801d31ebe98, State: Initialized, Role: FOLLOWER
I20260620 06:15:59.270290 5034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
I20260620 06:15:59.271494 5034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:15:59.272259 5185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
I20260620 06:15:59.272468 5185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280c432c047e4ca7ad1658f86f7ef14d, State: Initialized, Role: FOLLOWER
W20260620 06:15:59.272759 5206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 06:15:59.272753 5185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
W20260620 06:15:59.272861 5206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:59.273180 5217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.273363 5217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.273830 5185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:15:59.274106 4987 tablet.cc:2369] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:15:59.271165 5041 master_runner.cc:186] Error getting master registration for 127.14.209.190:42505: OK, instance_id { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" instance_seqno: 1781936158777919 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:15:59.274502 5008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 06:15:59.274544 5041 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:15:59.274608 5008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 0 FOLLOWER]: Advancing to term 1
W20260620 06:15:59.276393 5220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:59.276463 5220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:59.276842 5008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
I20260620 06:15:59.277530 5008 leader_election.cc:288] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [CANDIDATE]: Term 1 election: Requested vote from peers 0040927908894e02a415835d2e1fea0d (127.14.209.189:38947), 280c432c047e4ca7ad1658f86f7ef14d (127.14.209.188:36013)
I20260620 06:15:59.278049 5083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0040927908894e02a415835d2e1fea0d"
I20260620 06:15:59.278240 5083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:59.279495 5083 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e9e11ba67a43f9b5e3f801d31ebe98 in term 1.
I20260620 06:15:59.278258 5159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "280c432c047e4ca7ad1658f86f7ef14d"
I20260620 06:15:59.279884 4984 leader_election.cc:302] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0040927908894e02a415835d2e1fea0d, 26e9e11ba67a43f9b5e3f801d31ebe98; no voters:
I20260620 06:15:59.279983 5159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:15:59.280230 5207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.280340 5207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.280418 5207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:15:59.280475 5207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [term 1 LEADER]: Becoming Leader. State: Replica: 26e9e11ba67a43f9b5e3f801d31ebe98, State: Running, Role: LEADER
I20260620 06:15:59.280745 5207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } }
W20260620 06:15:59.281118 5232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:15:59.281183 5232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:15:59.281225 5159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e9e11ba67a43f9b5e3f801d31ebe98 in term 1.
I20260620 06:15:59.282330 5207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e9e11ba67a43f9b5e3f801d31ebe98. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.282429 5207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:59.285245 4966 master_runner.cc:263] No leader master found from master 26e9e11ba67a43f9b5e3f801d31ebe98
I20260620 06:15:59.285331 4966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:15:59.287807 5240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:15:59.288757 5240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:15:59.296010 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.282291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.14.209.189:38947)
0620 06:15:59.282403 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.282410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.287009 (+ 4599us) server_negotiation.cc:407] Connection header received
0620 06:15:59.287219 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.287223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.287272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.287324 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.288356 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.289055 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.290210 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.290533 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.290611 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.292478 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.292484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.292490 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.292517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.294257 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.294705 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.294709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.294711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.294788 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.295105 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.295108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.295110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.295323 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.295418 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.295760 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.295869 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:15:59.302798 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.285770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.14.209.190:42505)
0620 06:15:59.285872 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.285876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.290679 (+ 4803us) server_negotiation.cc:407] Connection header received
0620 06:15:59.290847 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.290850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.290892 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.290945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.293270 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.294098 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.297309 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.297558 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.297616 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.299301 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.299307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.299312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.299337 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.301089 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.301537 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.301540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.301542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.301620 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.301923 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.301926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.301927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.302115 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.302215 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.302537 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.302642 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:15:59.305723 5159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [term 1 FOLLOWER]: Refusing update from remote peer 26e9e11ba67a43f9b5e3f801d31ebe98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:59.306411 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:15:59.287212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.14.209.188:36013)
0620 06:15:59.287329 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.287334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.292133 (+ 4799us) server_negotiation.cc:407] Connection header received
0620 06:15:59.292296 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.292300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.292356 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.292408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.295251 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.295965 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.298009 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:15:59.298377 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:15:59.298463 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:15:59.302768 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:15:59.302772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:15:59.302774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:15:59.302798 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:15:59.304516 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.304999 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.305003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.305005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.305094 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:15:59.305404 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:15:59.305407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:15:59.305409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:15:59.305604 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:15:59.305723 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:15:59.306155 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 06:15:59.306266 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:15:59.306371 5207 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:59.310299 5217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e9e11ba67a43f9b5e3f801d31ebe98. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.310425 5217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.311728 5083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [term 1 FOLLOWER]: Refusing update from remote peer 26e9e11ba67a43f9b5e3f801d31ebe98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:15:59.312276 5238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:15:59.315382 5194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e9e11ba67a43f9b5e3f801d31ebe98. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.315503 5194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.319798 5238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.319911 5238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:59.320951 5245 mvcc.cc:205] Tried to move back new op lower bound from 7298810508500901888 to 7298810508416897024. Current Snapshot: MvccSnapshot[applied={T|T < 7298810508500901888}]
I20260620 06:15:59.321095 5217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.321188 5217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.321532 5244 mvcc.cc:205] Tried to move back new op lower bound from 7298810508500901888 to 7298810508416897024. Current Snapshot: MvccSnapshot[applied={T|T < 7298810508500901888}]
I20260620 06:15:59.326560 5207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.326670 5207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98 [sys.catalog]: This master's current role is: LEADER
I20260620 06:15:59.327603 5194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e9e11ba67a43f9b5e3f801d31ebe98" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 42505 } } peers { permanent_uuid: "0040927908894e02a415835d2e1fea0d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 38947 } } peers { permanent_uuid: "280c432c047e4ca7ad1658f86f7ef14d" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 36013 } } }
I20260620 06:15:59.327708 5194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:15:59.332684 5240 catalog_manager.cc:1347] Generated new cluster ID: 728c8d624dc645b1b8cbde76faa38cf0
I20260620 06:15:59.332751 5240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:15:59.355250 5240 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:15:59.356432 5240 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:15:59.369292 5240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26e9e11ba67a43f9b5e3f801d31ebe98: Generated new TSK 0
I20260620 06:15:59.370106 5240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:15:59.396880 5038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.394172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.14.209.190:42505)
0620 06:15:59.394280 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.394284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.395911 (+ 1627us) server_negotiation.cc:407] Connection header received
0620 06:15:59.396088 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.396091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.396135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.396184 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.396747 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36190: BlockingRecv error: recv got EOF from 127.0.0.1:36190 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 06:15:59.398293 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.394843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.14.209.189:38947)
0620 06:15:59.394943 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.394947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.397298 (+ 2351us) server_negotiation.cc:407] Connection header received
0620 06:15:59.397646 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.397650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.397694 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.397743 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.398188 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41452: BlockingRecv error: recv got EOF from 127.0.0.1:41452 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 06:15:59.399341 5188 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.395398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.14.209.188:36013)
0620 06:15:59.395494 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.395498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.397575 (+ 2077us) server_negotiation.cc:407] Connection header received
0620 06:15:59.398619 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.398622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.398664 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.398710 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.399233 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36098: BlockingRecv error: recv got EOF from 127.0.0.1:36098 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 06:15:59.403396 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.402012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.14.209.189:38947)
0620 06:15:59.402098 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.402102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.402525 (+ 423us) server_negotiation.cc:407] Connection header received
0620 06:15:59.402715 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.402718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.402791 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.402840 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.403307 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41468: BlockingRecv error: recv got EOF from 127.0.0.1:41468 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 06:15:59.407907 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.406568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.14.209.189:38947)
0620 06:15:59.406652 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.406655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.407149 (+ 494us) server_negotiation.cc:407] Connection header received
0620 06:15:59.407312 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.407315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.407355 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.407400 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.407829 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41472: BlockingRecv error: recv got EOF from 127.0.0.1:41472 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 06:15:59.412896 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.411628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.14.209.189:38947)
0620 06:15:59.411710 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.411714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.412166 (+ 452us) server_negotiation.cc:407] Connection header received
0620 06:15:59.412320 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.412323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.412362 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.412405 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.412818 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41478: BlockingRecv error: recv got EOF from 127.0.0.1:41478 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 06:15:59.420065 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.418555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.14.209.189:38947)
0620 06:15:59.418666 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.418670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.419234 (+ 564us) server_negotiation.cc:407] Connection header received
0620 06:15:59.419401 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.419404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.419445 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.419490 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.419988 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41486: BlockingRecv error: recv got EOF from 127.0.0.1:41486 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 06:15:59.431335 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.429868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.14.209.189:38947)
0620 06:15:59.429962 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.429966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.430459 (+ 493us) server_negotiation.cc:407] Connection header received
0620 06:15:59.430638 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.430643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.430688 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.430739 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.431241 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41492: BlockingRecv error: recv got EOF from 127.0.0.1:41492 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 06:15:59.450917 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.449320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.14.209.189:38947)
0620 06:15:59.449429 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.449434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.449985 (+ 551us) server_negotiation.cc:407] Connection header received
0620 06:15:59.450187 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.450191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.450241 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.450309 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.450821 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41502: BlockingRecv error: recv got EOF from 127.0.0.1:41502 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:15:59.486786 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.485622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.14.209.189:38947)
0620 06:15:59.485733 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.485737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.485920 (+ 183us) server_negotiation.cc:407] Connection header received
0620 06:15:59.486079 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.486083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.486138 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.486229 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.486660 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41504: BlockingRecv error: recv got EOF from 127.0.0.1:41504 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 06:15:59.554066 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.552901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.14.209.189:38947)
0620 06:15:59.553064 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.553069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.553219 (+ 150us) server_negotiation.cc:407] Connection header received
0620 06:15:59.553386 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.553390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.553444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.553536 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.553969 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41518: BlockingRecv error: recv got EOF from 127.0.0.1:41518 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 06:15:59.685184 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.684019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.14.209.189:38947)
0620 06:15:59.684154 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.684158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.684352 (+ 194us) server_negotiation.cc:407] Connection header received
0620 06:15:59.684465 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.684467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.684514 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.684592 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.685045 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41534: BlockingRecv error: recv got EOF from 127.0.0.1:41534 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 06:15:59.944517 5115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:15:59.943547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.14.209.189:38947)
0620 06:15:59.943688 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:15:59.943692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:15:59.943708 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:15:59.943854 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:15:59.943857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:15:59.943906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:15:59.943992 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:15:59.944373 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41538: BlockingRecv error: recv got EOF from 127.0.0.1:41538 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:16:00.274379 5206 catalog_manager.cc:1259] Loaded cluster ID: 728c8d624dc645b1b8cbde76faa38cf0
I20260620 06:16:00.274497 5206 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: loading cluster ID for follower catalog manager: success
I20260620 06:16:00.276389 5206 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: acquiring CA information for follower catalog manager: success
I20260620 06:16:00.277479 5206 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0040927908894e02a415835d2e1fea0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:16:00.282120 5232 catalog_manager.cc:1259] Loaded cluster ID: 728c8d624dc645b1b8cbde76faa38cf0
I20260620 06:16:00.282176 5232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: loading cluster ID for follower catalog manager: success
I20260620 06:16:00.283622 5232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: acquiring CA information for follower catalog manager: success
I20260620 06:16:00.284591 5232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 280c432c047e4ca7ad1658f86f7ef14d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:16:00.463977 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:00.459331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.14.209.189:38947)
0620 06:16:00.459739 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:00.459745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:00.459912 (+ 167us) server_negotiation.cc:407] Connection header received
0620 06:16:00.460311 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:00.460315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:00.460469 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:00.460603 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:00.460944 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.461497 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.462409 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.462638 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.462752 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:00.463121 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:00.463553 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:00.463692 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:00.463753 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 06:16:00.463811 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":93,"threads_started":1}
I20260620 06:16:00.471026 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:00.467131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.14.209.188:36013)
0620 06:16:00.467377 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:00.467381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:00.467394 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:16:00.467527 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:00.467529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:00.467633 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:00.467732 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:00.468217 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.468818 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.469737 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.469976 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.470056 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:00.470304 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:00.470682 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:00.470771 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:00.470844 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 06:16:00.470888 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260620 06:16:00.478246 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:00.474662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.14.209.189:38947)
0620 06:16:00.475123 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:00.475130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:00.475146 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:16:00.475189 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:00.475192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:00.475245 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:00.475365 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:00.475598 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.476366 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.477190 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.477424 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.477533 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:00.477813 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:00.477887 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:00.477964 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:00.478058 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 06:16:00.478103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260620 06:16:00.479597 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:00.475201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.14.209.188:36013)
0620 06:16:00.475333 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:00.475336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:00.475807 (+ 471us) server_negotiation.cc:407] Connection header received
0620 06:16:00.475894 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:00.475900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:00.475949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:00.476014 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:00.476324 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.477075 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.478453 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.478738 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.478792 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:00.479200 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:00.479282 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:00.479354 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:00.479439 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 06:16:00.479488 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:16:00.480073 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:00.474704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.14.209.190:42505)
0620 06:16:00.475565 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:00.475569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:00.475582 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:16:00.475676 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:00.475678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:00.475835 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:00.475945 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:00.476431 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.477229 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.478631 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:00.478957 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:00.479085 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:00.479385 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:00.479697 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:00.479788 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:00.479863 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 06:16:00.479914 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":67,"threads_started":1}
I20260620 06:16:00.483369 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 4966
I20260620 06:16:00.490000 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 5041
I20260620 06:16:00.495549 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 5116
2026-06-20T06:16:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3029 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:16:00 dist-test-slave-zt01 krb5kdc[5333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:16:00 dist-test-slave-zt01 krb5kdc[5333](info): set up 2 sockets
Jun 20 06:16:00 dist-test-slave-zt01 krb5kdc[5333](info): commencing operation
krb5kdc: starting...
W20260620 06:16:01.527377 15174 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/test-user.keytab.
Jun 20 06:16:01 dist-test-slave-zt01 krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.209.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
Entry for principal kudu/127.14.209.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab.
I20260620 06:16:01.645641 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:35431
--webserver_interface=127.14.209.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:35431,127.14.209.189:36899,127.14.209.188:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:16:01.749012 5346 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:16:01.749249 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:16:01.749271 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:16:01.749361 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:16:01.753423 5346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:16:01.753470 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:16:01.753489 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:16:01.753505 5346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:16:01.753563 5346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:16:01.758610 5346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:35431,127.14.209.189:36899,127.14.209.188:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.190.keytab
--principal=kudu/127.14.209.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.190:35431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.209.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:16:01.759531 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:16:01.760444 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:16:01.768394 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:16:01.768375 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:16:01.768501 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:16:01.768855 5346 server_base.cc:1034] running on GCE node
I20260620 06:16:01.769359 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:16:01.770335 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:16:01.771497 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781936161771471 us; error 36 us; skew 500 ppm
Jun 20 06:16:01 dist-test-slave-zt01 krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:16:01.777294 5346 init.cc:377] Logged in from keytab as kudu/127.14.209.190@KRBTEST.COM (short username kudu)
I20260620 06:16:01.778156 5346 webserver.cc:460] Webserver started at http://127.14.209.190:34371/ using document root <none> and password file <none>
I20260620 06:16:01.778674 5346 fs_manager.cc:359] Metadata directory not provided
I20260620 06:16:01.778765 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:16:01.778973 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:16:01.780817 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data/instance:
uuid: "3404cb576fef4c509539a6069ae6de58"
format_stamp: "Formatted at 2026-06-20 06:16:01 on dist-test-slave-zt01"
server_key: "f8766d3bf2cf48fc8763de7e392f3dd4"
server_key_iv: "b776e2895f5a59db9dbcd63219246447"
server_key_version: "encryptionkey@0"
I20260620 06:16:01.781337 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal/instance:
uuid: "3404cb576fef4c509539a6069ae6de58"
format_stamp: "Formatted at 2026-06-20 06:16:01 on dist-test-slave-zt01"
server_key: "f8766d3bf2cf48fc8763de7e392f3dd4"
server_key_iv: "b776e2895f5a59db9dbcd63219246447"
server_key_version: "encryptionkey@0"
I20260620 06:16:01.784703 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:16:01.786818 5361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:16:01.787932 5346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:16:01.788074 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal
uuid: "3404cb576fef4c509539a6069ae6de58"
format_stamp: "Formatted at 2026-06-20 06:16:01 on dist-test-slave-zt01"
server_key: "f8766d3bf2cf48fc8763de7e392f3dd4"
server_key_iv: "b776e2895f5a59db9dbcd63219246447"
server_key_version: "encryptionkey@0"
I20260620 06:16:01.788177 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:16:01.812191 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:16:01.815670 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:16:01.815878 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:16:01.822589 5346 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.190:35431
I20260620 06:16:01.822902 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.190:35431 every 8 connection(s)
I20260620 06:16:01.823495 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/data/info.pb
I20260620 06:16:01.826054 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:16:01.829613 5414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:01.830796 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 5346
I20260620 06:16:01.830914 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-0/wal/instance
I20260620 06:16:01.831251 15174 external_mini_cluster.cc:1428] Setting key d25c4711d8e562d6ad49f454130517fe
Jun 20 06:16:01 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.189@KRBTEST.COM" created.
I20260620 06:16:01.843839 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.831751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35431 (local address 127.0.0.1:59322)
0620 06:16:01.832220 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.832270 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:01.833074 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:01.833841 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:01.833846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:01.834143 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:01.834384 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:01.834396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:01.835589 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:01.835592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:01.836317 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:01.836326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:01.836964 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:01.836968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:01.837057 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:01.837617 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:01.837632 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:01.839325 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:01.841059 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:01.841067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:01.841073 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:01.841292 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:01.841665 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:01.841667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:01.841669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:01.841779 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:01.842402 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:01.842411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:01.842575 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 06:16:01.842759 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 06:16:01.843049 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":4,"thread_start_us":135,"threads_started":1}
I20260620 06:16:01.843844 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.831752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.14.209.190:35431)
0620 06:16:01.832311 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:01.832324 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:01.832369 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:16:01.833371 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:01.833378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:01.833622 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:01.833843 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:01.834536 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:01.835467 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:01.836443 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:01.836813 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:01.836978 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:01.839465 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:01.839470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:01.839478 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:01.839499 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:01.840946 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:01.841383 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:01.841387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:01.841388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:01.841530 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:01.841897 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:01.841904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:01.841908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:01.842275 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:01.842382 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:01.842815 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 06:16:01.842970 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":135,"threads_started":1}
I20260620 06:16:01.846123 5414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:01.846745 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.846525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35436)
0620 06:16:01.846631 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.846698 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:16:01.846956 5364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
I20260620 06:16:01.847712 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.847524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35450)
0620 06:16:01.847622 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.847670 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:16:01.847865 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:36899: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab.
Entry for principal kudu/127.14.209.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab.
I20260620 06:16:01.857527 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:36899
--webserver_interface=127.14.209.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab
--principal=kudu/127.14.209.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:35431,127.14.209.189:36899,127.14.209.188:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:16:01.897069 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } attempt: 1
I20260620 06:16:01.897794 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.897542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35456)
0620 06:16:01.897675 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.897727 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 06:16:01.898458 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.898254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35466)
0620 06:16:01.898361 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.898410 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:16:01.898574 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:36899: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
W20260620 06:16:01.960183 5421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:16:01.960479 5421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:16:01.960553 5421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:16:01.960633 5421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:16:01.964552 5421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:16:01.964617 5421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:16:01.964644 5421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:16:01.964668 5421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:16:01.964689 5421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:16:01.966739 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } attempt: 2
I20260620 06:16:01.967538 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.967266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35478)
0620 06:16:01.967406 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.967470 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 06:16:01.968217 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:01.968007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35482)
0620 06:16:01.968116 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:01.968165 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 06:16:01.968350 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.189:36899: Network error: Client connection negotiation failed: client connection to 127.14.209.189:36899: connect: Connection refused (error 111)
I20260620 06:16:01.970201 5421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:35431,127.14.209.189:36899,127.14.209.188:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.189.keytab
--principal=kudu/127.14.209.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.189:36899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.209.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:16:01.971174 5421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:16:01.972378 5421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:16:01.980417 5421 server_base.cc:1034] running on GCE node
W20260620 06:16:01.980441 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:16:01.980441 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:16:01.980556 5429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:16:01.981148 5421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:16:01.982070 5421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:16:01.983249 5421 hybrid_clock.cc:648] HybridClock initialized: now 1781936161983233 us; error 29 us; skew 500 ppm
Jun 20 06:16:01 dist-test-slave-zt01 krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:16:01.985775 5421 init.cc:377] Logged in from keytab as kudu/127.14.209.189@KRBTEST.COM (short username kudu)
I20260620 06:16:01.986567 5421 webserver.cc:460] Webserver started at http://127.14.209.189:39441/ using document root <none> and password file <none>
I20260620 06:16:01.987142 5421 fs_manager.cc:359] Metadata directory not provided
I20260620 06:16:01.987229 5421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:16:01.987447 5421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:16:01.989300 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data/instance:
uuid: "78e1bd49deee4ec6a726c20fe604758d"
format_stamp: "Formatted at 2026-06-20 06:16:01 on dist-test-slave-zt01"
server_key: "18756fa0fe99fb7b82333d37285ab9c9"
server_key_iv: "de9cdda1f3cb11943a6205f49f3e02a9"
server_key_version: "encryptionkey@0"
I20260620 06:16:01.989840 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal/instance:
uuid: "78e1bd49deee4ec6a726c20fe604758d"
format_stamp: "Formatted at 2026-06-20 06:16:01 on dist-test-slave-zt01"
server_key: "18756fa0fe99fb7b82333d37285ab9c9"
server_key_iv: "de9cdda1f3cb11943a6205f49f3e02a9"
server_key_version: "encryptionkey@0"
I20260620 06:16:01.993602 5421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:16:01.995885 5436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:16:01.997025 5421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:16:01.997255 5421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal
uuid: "78e1bd49deee4ec6a726c20fe604758d"
format_stamp: "Formatted at 2026-06-20 06:16:01 on dist-test-slave-zt01"
server_key: "18756fa0fe99fb7b82333d37285ab9c9"
server_key_iv: "de9cdda1f3cb11943a6205f49f3e02a9"
server_key_version: "encryptionkey@0"
I20260620 06:16:01.997368 5421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:16:02.023895 5421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:16:02.026480 5421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:16:02.026669 5421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:16:02.033294 5421 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.189:36899
I20260620 06:16:02.033572 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.189:36899 every 8 connection(s)
I20260620 06:16:02.034191 5421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/data/info.pb
I20260620 06:16:02.036988 5489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:16:02.040429 5489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:02.043028 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 5421
I20260620 06:16:02.043184 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-1/wal/instance
I20260620 06:16:02.043486 15174 external_mini_cluster.cc:1428] Setting key 325f458ad4b3d151a819171d027093e3
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.209.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.209.188@KRBTEST.COM" created.
I20260620 06:16:02.053897 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.042049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.14.209.190:35431)
0620 06:16:02.042208 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.042213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.042936 (+ 723us) server_negotiation.cc:407] Connection header received
0620 06:16:02.043961 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.043964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.044024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.044127 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.045162 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.045681 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.046779 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.047011 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.047166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.049426 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.049430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.049433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.049464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.051627 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.052207 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.052211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.052213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.052317 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.052728 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.052734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.052736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.052974 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.053094 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.053595 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.053718 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:16:02.054662 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.042383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35431 (local address 127.0.0.1:59328)
0620 06:16:02.042802 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.042840 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.043769 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.044293 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.044299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.044612 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.045021 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.045032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.045834 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.045837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.046647 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.046656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.047204 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.047208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.047366 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.047914 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.047929 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.049270 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.051755 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.051761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.051773 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.052107 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.052411 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.052414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.052416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.052548 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.053102 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.053110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.053357 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.053543 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.053828 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260620 06:16:02.056150 5489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:02.059527 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } attempt: 3
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
I20260620 06:16:02.067374 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.056634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35486)
0620 06:16:02.056768 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.056778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.056834 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.057341 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.057344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.057510 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.057576 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.057581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.058498 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.058504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.059004 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.059011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.059614 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.059617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.059750 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.060214 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.060227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.063533 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.065630 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.065634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.065637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.065920 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.066268 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.066274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.066279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.066382 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.066909 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.066914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.067015 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.067154 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.067260 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":38}
I20260620 06:16:02.067939 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.056741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.14.209.189:36899)
0620 06:16:02.057041 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.057049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.057064 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:16:02.057093 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.057096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.057220 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.057355 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.057682 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.058381 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.059153 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.059503 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.059616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.063669 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.063678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.063690 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.063713 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.065520 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.066045 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.066050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.066053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.066155 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.066509 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.066512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.066514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.066798 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.066904 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.067592 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.067743 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":226,"thread_start_us":130,"threads_started":1}
I20260620 06:16:02.068753 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.059907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35500)
0620 06:16:02.060026 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.060045 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.060119 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.060652 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.060657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.060797 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.060895 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.060900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.061641 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.061644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.062205 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.062213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.062742 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.062746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.062855 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.063393 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.063409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.064817 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.067196 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.067200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.067202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.067425 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.067752 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.067755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.067757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.067855 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.068256 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.068259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.068360 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.068464 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.068589 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 06:16:02.068810 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.060073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.14.209.189:36899)
0620 06:16:02.060393 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.060402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.060415 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:16:02.060453 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.060456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.060505 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.060597 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.060992 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.061551 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.062309 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.062623 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.062743 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.064938 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.064941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.064943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.064963 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.067009 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.067554 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.067557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.067558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.067636 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.067950 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.067952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.067953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.068169 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.068256 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.068520 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.068638 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.209.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab.
Entry for principal kudu/127.14.209.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab.
I20260620 06:16:02.069847 5489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:02.069983 5414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:02.070134 15174 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
/tmp/dist-test-taskepqMJ0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:46489
--webserver_interface=127.14.209.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab
--principal=kudu/127.14.209.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.209.148:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.209.190:35431,127.14.209.189:36899,127.14.209.188:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:16:02.070564 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.070236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36306)
0620 06:16:02.070410 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.070487 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 06:16:02.070586 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.070363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36318)
0620 06:16:02.070499 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.070538 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 06:16:02.070717 5440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
I20260620 06:16:02.071255 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.070995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36322)
0620 06:16:02.071136 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.071187 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 06:16:02.071466 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:46489: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
I20260620 06:16:02.071584 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.071307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36330)
0620 06:16:02.071475 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.071529 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 06:16:02.071713 5489 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:46489: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
I20260620 06:16:02.102614 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } attempt: 1
I20260620 06:16:02.103422 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.103086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36346)
0620 06:16:02.103264 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.103332 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 06:16:02.104288 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.104035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36356)
0620 06:16:02.104154 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.104205 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:16:02.104460 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:46489: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
I20260620 06:16:02.120949 5489 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } attempt: 1
I20260620 06:16:02.121591 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.121362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36372)
0620 06:16:02.121486 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.121534 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:16:02.122303 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.122124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36386)
0620 06:16:02.122213 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.122254 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 06:16:02.122495 5489 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:46489: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
W20260620 06:16:02.178860 5497 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:16:02.179172 5497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:16:02.179236 5497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:16:02.179311 5497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 06:16:02.179610 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } attempt: 2
I20260620 06:16:02.180410 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.180132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36390)
0620 06:16:02.180271 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.180341 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 06:16:02.181123 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.180900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36394)
0620 06:16:02.181019 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.181072 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 06:16:02.181243 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:46489: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
W20260620 06:16:02.183758 5497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:16:02.183835 5497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:16:02.183878 5497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:16:02.183919 5497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:16:02.183954 5497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:16:02.189030 5497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.209.148:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.209.190:35431,127.14.209.189:36899,127.14.209.188:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/krb5kdc/kudu_127.14.209.188.keytab
--principal=kudu/127.14.209.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.209.188:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.209.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:16:02.189967 5497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:16:02.190644 5489 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } attempt: 2
I20260620 06:16:02.190946 5497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:16:02.191395 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.191087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36408)
0620 06:16:02.191196 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.191287 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 06:16:02.192059 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.191822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36410)
0620 06:16:02.191928 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.191975 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:16:02.192209 5489 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.209.188:46489: Network error: Client connection negotiation failed: client connection to 127.14.209.188:46489: connect: Connection refused (error 111)
W20260620 06:16:02.199155 5502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:16:02.199193 5505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:16:02.199237 5503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:16:02.199321 5497 server_base.cc:1034] running on GCE node
I20260620 06:16:02.199908 5497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:16:02.200865 5497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:16:02.202052 5497 hybrid_clock.cc:648] HybridClock initialized: now 1781936162202025 us; error 37 us; skew 500 ppm
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:16:02.204794 5497 init.cc:377] Logged in from keytab as kudu/127.14.209.188@KRBTEST.COM (short username kudu)
I20260620 06:16:02.205775 5497 webserver.cc:460] Webserver started at http://127.14.209.188:43023/ using document root <none> and password file <none>
I20260620 06:16:02.206422 5497 fs_manager.cc:359] Metadata directory not provided
I20260620 06:16:02.206485 5497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:16:02.206631 5497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:16:02.208508 5497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data/instance:
uuid: "df91e6f64a514aa6bb8cedc3cf1f5039"
format_stamp: "Formatted at 2026-06-20 06:16:02 on dist-test-slave-zt01"
server_key: "5d80e1237366dd1b3493021c0b256948"
server_key_iv: "c907679b3a39669d69b02a557bf8e87f"
server_key_version: "encryptionkey@0"
I20260620 06:16:02.208963 5497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal/instance:
uuid: "df91e6f64a514aa6bb8cedc3cf1f5039"
format_stamp: "Formatted at 2026-06-20 06:16:02 on dist-test-slave-zt01"
server_key: "5d80e1237366dd1b3493021c0b256948"
server_key_iv: "c907679b3a39669d69b02a557bf8e87f"
server_key_version: "encryptionkey@0"
I20260620 06:16:02.212206 5497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:16:02.214783 5512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:16:02.216050 5497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:16:02.216197 5497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data,/tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal
uuid: "df91e6f64a514aa6bb8cedc3cf1f5039"
format_stamp: "Formatted at 2026-06-20 06:16:02 on dist-test-slave-zt01"
server_key: "5d80e1237366dd1b3493021c0b256948"
server_key_iv: "c907679b3a39669d69b02a557bf8e87f"
server_key_version: "encryptionkey@0"
I20260620 06:16:02.216317 5497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:16:02.229974 5497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:16:02.232486 5497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:16:02.232664 5497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:16:02.239037 5497 rpc_server.cc:307] RPC server started. Bound to: 127.14.209.188:46489
I20260620 06:16:02.239341 5564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.209.188:46489 every 8 connection(s)
I20260620 06:16:02.239928 5497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/data/info.pb
I20260620 06:16:02.242379 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:16:02.245942 15174 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu as pid 5497
I20260620 06:16:02.245788 5565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:02.246085 15174 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskepqMJ0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935885709390-15174-0/minicluster-data/master-2/wal/instance
I20260620 06:16:02.246357 15174 external_mini_cluster.cc:1428] Setting key 77aacb09594cf7311eb92836210f4362
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:16:02.258822 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.247552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.14.209.190:35431)
0620 06:16:02.247697 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.247702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.248116 (+ 414us) server_negotiation.cc:407] Connection header received
0620 06:16:02.248888 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.248891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.248938 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.249024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.250332 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.251080 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.251971 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.252214 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.252326 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.255081 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.255087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.255090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.255121 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.256745 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.257306 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.257309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.257310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.257380 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.257718 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.257721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.257723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.257941 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.258038 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.258521 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.258660 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:16:02.259526 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.247548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.14.209.188:46489)
0620 06:16:02.247981 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.247994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.248022 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:16:02.248656 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.248671 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.248900 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.249059 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.249715 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.250887 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.251770 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.252145 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.252445 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.255561 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.255566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.255574 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.255593 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.257033 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.257498 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.257503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.257505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.257637 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.257938 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.257941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.257942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.258257 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.258392 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.258670 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.258798 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":17536,"thread_start_us":182,"threads_started":1}
I20260620 06:16:02.259523 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.247548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.190:35431 (local address 127.0.0.1:59340)
0620 06:16:02.247978 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.248021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.248674 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.249773 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.249777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.250052 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.250194 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.250204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.251171 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.251175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.251849 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.251858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.252351 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.252355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.252462 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.253141 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.253187 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.254909 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.256852 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.256862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.256870 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.257194 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.257489 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.257492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.257494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.257628 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.258044 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.258048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.258269 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.258445 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.258716 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":34,"spinlock_wait_cycles":2944,"thread_start_us":121,"threads_started":1}
I20260620 06:16:02.260759 5565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.190@KRBTEST.COM
I20260620 06:16:02.268512 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.261168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.189:36899 (local address 127.0.0.1:35510)
0620 06:16:02.261321 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.261333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.261398 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.261589 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.261591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.261704 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.261823 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.261832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.262604 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.262607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.263161 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.263169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.263689 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.263692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.263783 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.264137 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.264154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.265298 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.267110 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.267113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.267116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.267349 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.267631 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.267634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.267636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.267714 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.268071 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.268074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.268154 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.268261 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.268372 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 06:16:02.268568 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.261242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.14.209.189:36899)
0620 06:16:02.261362 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.261366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.261381 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:16:02.261467 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.261470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.261521 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.261583 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.261991 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.262515 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.263270 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.263591 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.263667 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.265446 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.265451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.265455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.265481 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.266969 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.267452 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.267455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.267457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.267534 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.267811 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.267814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.267816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.267977 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.268082 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.268313 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.268427 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:16:02.269150 5565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:16:02.270408 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.261918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.14.209.190:35431)
0620 06:16:02.262655 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.262658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.262666 (+ 8us) server_negotiation.cc:407] Connection header received
0620 06:16:02.262692 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.262694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.262725 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.262812 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.263454 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.264057 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.264802 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.265005 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.265115 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.267303 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.267306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.267308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.267326 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.268799 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.269232 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.269235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.269243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.269324 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.269628 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.269632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.269633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.269798 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.269995 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.270145 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.270260 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676}
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.188@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.209.189@KRBTEST.COM
I20260620 06:16:02.277549 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.269604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.14.209.188:46489)
0620 06:16:02.269698 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.269702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.269713 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:16:02.269800 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.269805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.269836 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.269914 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.270220 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.270930 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.271909 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.272111 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.272184 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.274010 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.274014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.274016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.274036 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.275679 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.276097 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.276100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.276101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.276165 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.276712 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.276717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.276719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.276935 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.277009 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.277309 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.277424 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:16:02.277578 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.269552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36434)
0620 06:16:02.269641 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.269655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.269720 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.269931 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.269934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.270074 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.270124 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.270128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.271033 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.271039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.271799 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.271809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.272213 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.272217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.272292 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.272725 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.272745 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.273886 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.275822 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.275825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.275826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.275995 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.276256 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.276259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.276261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.276349 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.277033 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.277038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.277135 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.277243 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.277350 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 06:16:02.279781 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.271473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.14.209.189:36899)
0620 06:16:02.271574 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.271577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.271590 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:16:02.271667 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.271669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.271698 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.271762 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.272385 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.272817 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.273647 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.273892 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.274002 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.276476 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.276478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.276480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.276498 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.278293 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.278720 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.278723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.278724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.278789 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.279041 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.279043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.279044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.279234 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.279367 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.279599 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.279681 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:16:02.280390 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } attempt: 3
I20260620 06:16:02.281415 5565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: Bootstrap starting.
I20260620 06:16:02.283157 15174 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:16:02.283376 5489 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } attempt: 3
I20260620 06:16:02.283934 5565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: Neither blocks nor log segments found. Creating new log.
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.190@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:16:02.284327 5388 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3404cb576fef4c509539a6069ae6de58"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59356
I20260620 06:16:02.286279 5565 log.cc:824] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: Log is configured to *not* fsync() on all Append() calls
I20260620 06:16:02.288537 5565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: No bootstrap required, opened a new log
I20260620 06:16:02.289217 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.280796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36444)
0620 06:16:02.280910 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.280924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.281000 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.281262 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.281265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.281422 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.281511 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.281518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.282237 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.282241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.282999 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.283010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.283568 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.283571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.283666 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.284246 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.284261 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.285614 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.287501 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.287505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.287507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.287695 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.287975 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.287978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.287980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.288075 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.288715 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.288719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.288824 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.288904 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.289045 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
Jun 20 06:16:02 dist-test-slave-zt01 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781936161, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.209.189@KRBTEST.COM for kudu/127.14.209.188@KRBTEST.COM
I20260620 06:16:02.290004 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.280835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.14.209.188:46489)
0620 06:16:02.280917 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.280921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.280965 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:16:02.281086 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.281089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.281149 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.281221 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.281608 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.282127 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.283143 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.283443 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.283903 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.285721 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.285724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.285727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.285758 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.287416 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.287797 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.287802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.287804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.287890 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.288313 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.288316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.288318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.288535 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.289779 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.289845 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.289920 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 06:16:02.292704 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: Bootstrap starting.
I20260620 06:16:02.292642 5565 raft_consensus.cc:348] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.292845 5565 raft_consensus.cc:729] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df91e6f64a514aa6bb8cedc3cf1f5039, State: Initialized, Role: FOLLOWER
I20260620 06:16:02.293159 5565 consensus_queue.cc:260] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.293720 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.284141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.209.188:46489 (local address 127.0.0.1:36452)
0620 06:16:02.284250 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:16:02.284266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:16:02.284341 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:16:02.284813 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:16:02.284818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:16:02.284983 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:16:02.285087 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.285093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.286302 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.286306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.287407 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:16:02.287418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.288029 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.288035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:16:02.288132 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.288569 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:16:02.288589 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:16:02.289946 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:16:02.292232 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.292235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.292237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.292469 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.292796 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:16:02.292799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:16:02.292800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:16:02.292884 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:16:02.293256 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:16:02.293258 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:16:02.293343 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 06:16:02.293446 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 06:16:02.293559 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 06:16:02.293778 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.284341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.14.209.188:46489)
0620 06:16:02.284602 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.284610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.284622 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:16:02.284651 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.284653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.284695 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.284788 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.285201 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.285914 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.287544 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.287889 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.288023 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.290275 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.290278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.290282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.290336 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.292105 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.292587 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.292590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.292592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.292692 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.292981 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.292985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.292986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.293161 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.293259 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.293514 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.293615 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260620 06:16:02.294075 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.294178 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:16:02.294392 5565 sys_catalog.cc:564] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:16:02.295001 5414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: Neither blocks nor log segments found. Creating new log.
I20260620 06:16:02.295784 5414 log.cc:824] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: Log is configured to *not* fsync() on all Append() calls
I20260620 06:16:02.296403 5489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: Bootstrap starting.
I20260620 06:16:02.297686 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: No bootstrap required, opened a new log
I20260620 06:16:02.298523 5489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: Neither blocks nor log segments found. Creating new log.
I20260620 06:16:02.299316 5489 log.cc:824] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:16:02.300503 5414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.300729 5414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3404cb576fef4c509539a6069ae6de58, State: Initialized, Role: FOLLOWER
I20260620 06:16:02.301035 5414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.301514 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.301673 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:16:02.302177 5414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:16:02.300335 5497 master_runner.cc:186] Error getting master registration for 127.14.209.190:35431: OK, instance_id { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" instance_seqno: 1781936161820678 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:16:02.303494 5497 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:16:02.303610 5388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 06:16:02.303707 5388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:16:02.304574 5489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: No bootstrap required, opened a new log
I20260620 06:16:02.304798 5388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.305689 5388 leader_election.cc:288] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [CANDIDATE]: Term 1 election: Requested vote from peers 78e1bd49deee4ec6a726c20fe604758d (127.14.209.189:36899), df91e6f64a514aa6bb8cedc3cf1f5039 (127.14.209.188:46489)
I20260620 06:16:02.306200 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3404cb576fef4c509539a6069ae6de58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039"
I20260620 06:16:02.306397 5539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:16:02.308418 5489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.308615 5489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e1bd49deee4ec6a726c20fe604758d, State: Initialized, Role: FOLLOWER
I20260620 06:16:02.308914 5489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.310515 5489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:16:02.309407 5346 master_runner.cc:186] Error getting master registration for 127.14.209.189:36899: OK, instance_id { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" instance_seqno: 1781936162031509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:16:02.313046 5346 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:16:02.313141 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3404cb576fef4c509539a6069ae6de58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "78e1bd49deee4ec6a726c20fe604758d"
I20260620 06:16:02.313334 5462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:16:02.314481 5462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3404cb576fef4c509539a6069ae6de58 in term 1.
I20260620 06:16:02.314832 5364 leader_election.cc:302] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3404cb576fef4c509539a6069ae6de58, 78e1bd49deee4ec6a726c20fe604758d; no voters:
I20260620 06:16:02.315104 5587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:16:02.315312 5587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [term 1 LEADER]: Becoming Leader. State: Replica: 3404cb576fef4c509539a6069ae6de58, State: Running, Role: LEADER
I20260620 06:16:02.315603 5587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } }
I20260620 06:16:02.317148 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3404cb576fef4c509539a6069ae6de58. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.317286 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: This master's current role is: LEADER
I20260620 06:16:02.317333 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.317467 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: This master's current role is: FOLLOWER
W20260620 06:16:02.317523 5367 tablet.cc:2369] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:16:02.318833 5539 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3404cb576fef4c509539a6069ae6de58 in term 1.
W20260620 06:16:02.298862 5584 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:16:02.319790 5584 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:16:02.320966 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.308099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.14.209.190:35431)
0620 06:16:02.308198 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.308202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.309658 (+ 1456us) server_negotiation.cc:407] Connection header received
0620 06:16:02.309827 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.309830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.309872 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.309920 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.312340 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.313110 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.314014 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.314322 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.314413 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.316432 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.316436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.316439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.316462 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.318560 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.319334 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.319337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.319339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.319426 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.319976 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.319979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.319981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.320198 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.320337 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.320685 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.320803 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 06:16:02.322626 5598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:16:02.322690 5598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:16:02.324100 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:16:02.324839 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:16:02.327205 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.308682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.14.209.188:46489)
0620 06:16:02.308794 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.308798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.311021 (+ 2223us) server_negotiation.cc:407] Connection header received
0620 06:16:02.311217 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.311221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.311298 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.311355 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.314059 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.314746 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.316217 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.316592 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.316701 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.318572 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.318576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.318581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.318609 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.320798 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.325786 (+ 4988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.325795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.325799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.325889 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.326207 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.326212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.326214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.326451 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.326565 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.326886 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.327003 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 06:16:02.329322 5617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:16:02.329401 5617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:16:02.330263 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:02.309190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.14.209.189:36899)
0620 06:16:02.309287 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.309294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.322194 (+ 12900us) server_negotiation.cc:407] Connection header received
0620 06:16:02.322355 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.322358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.322401 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.322458 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.323210 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.323878 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.324864 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:02.325193 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:02.325261 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:02.326827 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:16:02.326832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:16:02.326835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:16:02.326858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:16:02.328639 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.329052 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.329056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.329058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.329151 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:16:02.329491 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:16:02.329494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:16:02.329496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:16:02.329653 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:16:02.329780 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:02.330007 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 06:16:02.330114 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:16:02.333707 5539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [term 1 FOLLOWER]: Refusing update from remote peer 3404cb576fef4c509539a6069ae6de58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:16:02.333737 5462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [term 1 FOLLOWER]: Refusing update from remote peer 3404cb576fef4c509539a6069ae6de58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:16:02.334327 5587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:16:02.334697 5587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:16:02.338191 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3404cb576fef4c509539a6069ae6de58. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.338299 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:16:02.338990 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3404cb576fef4c509539a6069ae6de58. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.339109 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:16:02.341272 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.341373 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: This master's current role is: LEADER
I20260620 06:16:02.341518 5622 mvcc.cc:205] Tried to move back new op lower bound from 7298810520913219584 to 7298810520847708160. Current Snapshot: MvccSnapshot[applied={T|T < 7298810520913219584}]
I20260620 06:16:02.341773 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.341871 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:16:02.343335 5625 mvcc.cc:205] Tried to move back new op lower bound from 7298810520913219584 to 7298810520847708160. Current Snapshot: MvccSnapshot[applied={T|T < 7298810520913219584}]
I20260620 06:16:02.348659 5618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.348779 5618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58 [sys.catalog]: This master's current role is: LEADER
I20260620 06:16:02.349628 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3404cb576fef4c509539a6069ae6de58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3404cb576fef4c509539a6069ae6de58" member_type: VOTER last_known_addr { host: "127.14.209.190" port: 35431 } } peers { permanent_uuid: "78e1bd49deee4ec6a726c20fe604758d" member_type: VOTER last_known_addr { host: "127.14.209.189" port: 36899 } } peers { permanent_uuid: "df91e6f64a514aa6bb8cedc3cf1f5039" member_type: VOTER last_known_addr { host: "127.14.209.188" port: 46489 } } }
I20260620 06:16:02.349746 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:16:02.351218 5624 mvcc.cc:205] Tried to move back new op lower bound from 7298810520913219584 to 7298810520847708160. Current Snapshot: MvccSnapshot[applied={T|T < 7298810520913219584}]
I20260620 06:16:02.351415 5620 catalog_manager.cc:1347] Generated new cluster ID: e182ba86d0b24507aeb6b9159d65dde1
I20260620 06:16:02.351471 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:16:02.389372 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:16:02.390553 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:16:02.402757 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3404cb576fef4c509539a6069ae6de58: Generated new TSK 0
I20260620 06:16:02.403455 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:16:02.431988 5417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.429501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.14.209.190:35431)
0620 06:16:02.429611 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.429615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.430972 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 06:16:02.431175 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.431179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.431228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.431279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.431861 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59386: BlockingRecv error: recv got EOF from 127.0.0.1:59386 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 06:16:02.433422 5570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.430123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.14.209.188:46489)
0620 06:16:02.430238 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.430242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.432403 (+ 2161us) server_negotiation.cc:407] Connection header received
0620 06:16:02.432764 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.432767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.432811 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.432859 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.433316 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36480: BlockingRecv error: recv got EOF from 127.0.0.1:36480 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 06:16:02.434438 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.430640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.14.209.189:36899)
0620 06:16:02.430744 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.430748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.432590 (+ 1842us) server_negotiation.cc:407] Connection header received
0620 06:16:02.433747 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.433751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.433796 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.433846 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.434332 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35538: BlockingRecv error: recv got EOF from 127.0.0.1:35538 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 06:16:02.438000 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.436600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.14.209.189:36899)
0620 06:16:02.436690 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.436694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.437141 (+ 447us) server_negotiation.cc:407] Connection header received
0620 06:16:02.437338 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.437342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.437412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.437461 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.437913 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35548: BlockingRecv error: recv got EOF from 127.0.0.1:35548 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 06:16:02.442360 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.441101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.14.209.189:36899)
0620 06:16:02.441188 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.441191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.441595 (+ 404us) server_negotiation.cc:407] Connection header received
0620 06:16:02.441771 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.441774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.441816 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.441864 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.442283 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35556: BlockingRecv error: recv got EOF from 127.0.0.1:35556 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 06:16:02.447481 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.446146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.14.209.189:36899)
0620 06:16:02.446232 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.446235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.446691 (+ 456us) server_negotiation.cc:407] Connection header received
0620 06:16:02.446868 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.446872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.446911 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.446956 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.447406 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35564: BlockingRecv error: recv got EOF from 127.0.0.1:35564 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 06:16:02.454677 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.453337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.14.209.189:36899)
0620 06:16:02.453445 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.453448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.453923 (+ 475us) server_negotiation.cc:407] Connection header received
0620 06:16:02.454079 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.454082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.454121 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.454165 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.454580 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35568: BlockingRecv error: recv got EOF from 127.0.0.1:35568 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 06:16:02.466022 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.464587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.14.209.189:36899)
0620 06:16:02.464681 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.464684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.465185 (+ 501us) server_negotiation.cc:407] Connection header received
0620 06:16:02.465360 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.465366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.465409 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.465460 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.465935 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35576: BlockingRecv error: recv got EOF from 127.0.0.1:35576 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 06:16:02.485867 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.484221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.14.209.189:36899)
0620 06:16:02.484330 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.484334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.484886 (+ 552us) server_negotiation.cc:407] Connection header received
0620 06:16:02.485088 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.485092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.485144 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.485197 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.485754 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35582: BlockingRecv error: recv got EOF from 127.0.0.1:35582 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 06:16:02.521530 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.520258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.14.209.189:36899)
0620 06:16:02.520375 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.520386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.520696 (+ 310us) server_negotiation.cc:407] Connection header received
0620 06:16:02.520854 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.520857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.520916 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.521004 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.521414 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35586: BlockingRecv error: recv got EOF from 127.0.0.1:35586 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 06:16:02.588642 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.587471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.14.209.189:36899)
0620 06:16:02.587633 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.587638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.587793 (+ 155us) server_negotiation.cc:407] Connection header received
0620 06:16:02.587952 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.587956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.588009 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.588094 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.588528 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35592: BlockingRecv error: recv got EOF from 127.0.0.1:35592 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 06:16:02.719596 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.718476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.14.209.189:36899)
0620 06:16:02.718636 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.718640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.718705 (+ 65us) server_negotiation.cc:407] Connection header received
0620 06:16:02.718832 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.718835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.718884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.718957 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.719471 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35606: BlockingRecv error: recv got EOF from 127.0.0.1:35606 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 06:16:02.978685 5495 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:16:02.977550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.14.209.189:36899)
0620 06:16:02.977696 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:02.977703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:02.977777 (+ 74us) server_negotiation.cc:407] Connection header received
0620 06:16:02.977977 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:02.977980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:02.978037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:02.978128 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:16:02.978571 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35612: BlockingRecv error: recv got EOF from 127.0.0.1:35612 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:16:03.321133 5584 catalog_manager.cc:1259] Loaded cluster ID: e182ba86d0b24507aeb6b9159d65dde1
I20260620 06:16:03.321218 5584 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: loading cluster ID for follower catalog manager: success
I20260620 06:16:03.322777 5584 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: acquiring CA information for follower catalog manager: success
I20260620 06:16:03.323917 5584 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df91e6f64a514aa6bb8cedc3cf1f5039: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:16:03.330360 5617 catalog_manager.cc:1259] Loaded cluster ID: e182ba86d0b24507aeb6b9159d65dde1
I20260620 06:16:03.330412 5617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: loading cluster ID for follower catalog manager: success
I20260620 06:16:03.331876 5617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: acquiring CA information for follower catalog manager: success
I20260620 06:16:03.332835 5617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78e1bd49deee4ec6a726c20fe604758d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:16:03.496626 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:03.492675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.14.209.189:36899)
0620 06:16:03.492928 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:03.492933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:03.492968 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:16:03.493112 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:03.493115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:03.493231 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:03.493326 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:03.493544 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.494129 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.495154 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.495369 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.495459 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:03.495762 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:03.496191 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:03.496284 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:03.496364 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 06:16:03.496420 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260620 06:16:03.502887 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:03.499368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.14.209.188:46489)
0620 06:16:03.499573 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:03.499576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:03.499589 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:16:03.499654 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:03.499656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:03.499745 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:03.499834 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:03.500062 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.500641 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.501466 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.501790 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.501892 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:03.502172 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:03.502480 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:03.502602 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:03.502644 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 06:16:03.502738 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260620 06:16:03.509608 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:03.506242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.14.209.188:46489)
0620 06:16:03.506368 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:03.506372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:03.506570 (+ 198us) server_negotiation.cc:407] Connection header received
0620 06:16:03.506724 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:03.506728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:03.506774 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:03.506844 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:03.507152 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.507616 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.508544 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.508844 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.508944 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:03.509176 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:03.509248 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:03.509294 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:03.509452 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 06:16:03.509494 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 06:16:03.510018 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:03.505738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.14.209.189:36899)
0620 06:16:03.505893 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:03.505897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:03.506027 (+ 130us) server_negotiation.cc:407] Connection header received
0620 06:16:03.506193 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:03.506196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:03.506243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:03.506310 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:03.506666 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.507433 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.508694 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.509036 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.509142 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:03.509436 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:03.509514 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:03.509561 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:03.509831 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 06:16:03.509878 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 06:16:03.510651 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:16:03.505744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.14.209.190:35431)
0620 06:16:03.505965 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 06:16:03.505970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:16:03.505986 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:16:03.506148 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:16:03.506151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:16:03.506438 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:16:03.506578 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:16:03.506866 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.507641 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.508867 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:16:03.509214 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:16:03.509321 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:16:03.509641 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:16:03.509903 (+ 262us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:16:03.509977 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:16:03.510440 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 06:16:03.510521 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260620 06:16:03.513456 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 5346
I20260620 06:16:03.519613 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 5421
I20260620 06:16:03.525094 15174 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskepqMJ0/build/debug/bin/kudu with pid 5497
2026-06-20T06:16:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3029 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6058 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277805 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST